Behavior of agreement measures in the presence of zero cells and biased marginal distributions

Shankar Viswanathan, Shrikant I. Bangdiwala

Research output: Contribution to journalArticle

6 Citations (Scopus)

Abstract

Kappa and B assess agreement between two observers independently classifying N units into k categories. We study their behavior under zero cells in the contingency table and unbalanced asymmetric marginal distributions. Zero cells arise when a cross-classification is never endorsed by both observers; biased marginal distributions occur when some categories are preferred differently between the observers. Simulations studied the distributions of the unweighted and weighted statistics for k = 4, under fixed proportions of diagonal agreement and different patterns off-diagonal, with various sample sizes, and under various zero cell count scenarios. Marginal distributions were first uniform and homogeneous, and then unbalanced asymmetric distributions. Results for unweighted kappa and B statistics were comparable to work of Muñoz and Bangdiwala, even with zero cells. A slight increased variation was observed as the sample size decreased. Weighted statistics did show greater variation as the number of zero cells increased, with weighted kappa increasing substantially more than weighted B. Under biased marginal distributions, weighted kappa with Cicchetti weights were higher than with squared weights. Both statistics for observer agreement behaved well under zero cells. The weighted B was less variable than the weighted kappa under similar circumstances and different weights. In general, B's performance and graphical interpretation make it preferable to kappa under the studied scenarios.

Original languageEnglish (US)
Pages (from-to)445-464
Number of pages20
JournalJournal of Applied Statistics
Volume35
Issue number4
DOIs
StatePublished - Apr 2008
Externally publishedYes

Fingerprint

Marginal Distribution
Biased
Observer
Cell
Zero
Statistics
Asymmetric Distribution
Sample Size
Scenarios
Contingency Table
Count
Proportion
Unit
Simulation

Keywords

  • Bangdiwala's B
  • Cohen's kappa
  • Observer bias
  • Zero cell

ASJC Scopus subject areas

  • Statistics and Probability

Cite this

Behavior of agreement measures in the presence of zero cells and biased marginal distributions. / Viswanathan, Shankar; Bangdiwala, Shrikant I.

In: Journal of Applied Statistics, Vol. 35, No. 4, 04.2008, p. 445-464.

Research output: Contribution to journalArticle

@article{c9164c7e7c8b4ec195567a17b49b8b00,
title = "Behavior of agreement measures in the presence of zero cells and biased marginal distributions",
abstract = "Kappa and B assess agreement between two observers independently classifying N units into k categories. We study their behavior under zero cells in the contingency table and unbalanced asymmetric marginal distributions. Zero cells arise when a cross-classification is never endorsed by both observers; biased marginal distributions occur when some categories are preferred differently between the observers. Simulations studied the distributions of the unweighted and weighted statistics for k = 4, under fixed proportions of diagonal agreement and different patterns off-diagonal, with various sample sizes, and under various zero cell count scenarios. Marginal distributions were first uniform and homogeneous, and then unbalanced asymmetric distributions. Results for unweighted kappa and B statistics were comparable to work of Mu{\~n}oz and Bangdiwala, even with zero cells. A slight increased variation was observed as the sample size decreased. Weighted statistics did show greater variation as the number of zero cells increased, with weighted kappa increasing substantially more than weighted B. Under biased marginal distributions, weighted kappa with Cicchetti weights were higher than with squared weights. Both statistics for observer agreement behaved well under zero cells. The weighted B was less variable than the weighted kappa under similar circumstances and different weights. In general, B's performance and graphical interpretation make it preferable to kappa under the studied scenarios.",
keywords = "Bangdiwala's B, Cohen's kappa, Observer bias, Zero cell",
author = "Shankar Viswanathan and Bangdiwala, {Shrikant I.}",
year = "2008",
month = "4",
doi = "10.1080/02664760701835052",
language = "English (US)",
volume = "35",
pages = "445--464",
journal = "Journal of Applied Statistics",
issn = "0266-4763",
publisher = "Routledge",
number = "4",

}

TY - JOUR

T1 - Behavior of agreement measures in the presence of zero cells and biased marginal distributions

AU - Viswanathan, Shankar

AU - Bangdiwala, Shrikant I.

PY - 2008/4

Y1 - 2008/4

N2 - Kappa and B assess agreement between two observers independently classifying N units into k categories. We study their behavior under zero cells in the contingency table and unbalanced asymmetric marginal distributions. Zero cells arise when a cross-classification is never endorsed by both observers; biased marginal distributions occur when some categories are preferred differently between the observers. Simulations studied the distributions of the unweighted and weighted statistics for k = 4, under fixed proportions of diagonal agreement and different patterns off-diagonal, with various sample sizes, and under various zero cell count scenarios. Marginal distributions were first uniform and homogeneous, and then unbalanced asymmetric distributions. Results for unweighted kappa and B statistics were comparable to work of Muñoz and Bangdiwala, even with zero cells. A slight increased variation was observed as the sample size decreased. Weighted statistics did show greater variation as the number of zero cells increased, with weighted kappa increasing substantially more than weighted B. Under biased marginal distributions, weighted kappa with Cicchetti weights were higher than with squared weights. Both statistics for observer agreement behaved well under zero cells. The weighted B was less variable than the weighted kappa under similar circumstances and different weights. In general, B's performance and graphical interpretation make it preferable to kappa under the studied scenarios.

AB - Kappa and B assess agreement between two observers independently classifying N units into k categories. We study their behavior under zero cells in the contingency table and unbalanced asymmetric marginal distributions. Zero cells arise when a cross-classification is never endorsed by both observers; biased marginal distributions occur when some categories are preferred differently between the observers. Simulations studied the distributions of the unweighted and weighted statistics for k = 4, under fixed proportions of diagonal agreement and different patterns off-diagonal, with various sample sizes, and under various zero cell count scenarios. Marginal distributions were first uniform and homogeneous, and then unbalanced asymmetric distributions. Results for unweighted kappa and B statistics were comparable to work of Muñoz and Bangdiwala, even with zero cells. A slight increased variation was observed as the sample size decreased. Weighted statistics did show greater variation as the number of zero cells increased, with weighted kappa increasing substantially more than weighted B. Under biased marginal distributions, weighted kappa with Cicchetti weights were higher than with squared weights. Both statistics for observer agreement behaved well under zero cells. The weighted B was less variable than the weighted kappa under similar circumstances and different weights. In general, B's performance and graphical interpretation make it preferable to kappa under the studied scenarios.

KW - Bangdiwala's B

KW - Cohen's kappa

KW - Observer bias

KW - Zero cell

UR - http://www.scopus.com/inward/record.url?scp=41449119287&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=41449119287&partnerID=8YFLogxK

U2 - 10.1080/02664760701835052

DO - 10.1080/02664760701835052

M3 - Article

VL - 35

SP - 445

EP - 464

JO - Journal of Applied Statistics

JF - Journal of Applied Statistics

SN - 0266-4763

IS - 4

ER -