Abstract
Kappa and B assess agreement between two observers independently classifying N units into k categories. We study their behavior under zero cells in the contingency table and unbalanced asymmetric marginal distributions. Zero cells arise when a cross-classification is never endorsed by both observers; biased marginal distributions occur when some categories are preferred differently between the observers. Simulations studied the distributions of the unweighted and weighted statistics for k = 4, under fixed proportions of diagonal agreement and different patterns off-diagonal, with various sample sizes, and under various zero cell count scenarios. Marginal distributions were first uniform and homogeneous, and then unbalanced asymmetric distributions. Results for unweighted kappa and B statistics were comparable to work of Muñoz and Bangdiwala, even with zero cells. A slight increased variation was observed as the sample size decreased. Weighted statistics did show greater variation as the number of zero cells increased, with weighted kappa increasing substantially more than weighted B. Under biased marginal distributions, weighted kappa with Cicchetti weights were higher than with squared weights. Both statistics for observer agreement behaved well under zero cells. The weighted B was less variable than the weighted kappa under similar circumstances and different weights. In general, B's performance and graphical interpretation make it preferable to kappa under the studied scenarios.
| Original language | English (US) |
|---|---|
| Pages (from-to) | 445-464 |
| Number of pages | 20 |
| Journal | Journal of Applied Statistics |
| Volume | 35 |
| Issue number | 4 |
| DOIs | |
| State | Published - Apr 2008 |
| Externally published | Yes |
Fingerprint
Keywords
- Bangdiwala's B
- Cohen's kappa
- Observer bias
- Zero cell
ASJC Scopus subject areas
- Statistics and Probability
Cite this
Behavior of agreement measures in the presence of zero cells and biased marginal distributions. / Viswanathan, Shankar; Bangdiwala, Shrikant I.
In: Journal of Applied Statistics, Vol. 35, No. 4, 04.2008, p. 445-464.Research output: Contribution to journal › Article
}
TY - JOUR
T1 - Behavior of agreement measures in the presence of zero cells and biased marginal distributions
AU - Viswanathan, Shankar
AU - Bangdiwala, Shrikant I.
PY - 2008/4
Y1 - 2008/4
N2 - Kappa and B assess agreement between two observers independently classifying N units into k categories. We study their behavior under zero cells in the contingency table and unbalanced asymmetric marginal distributions. Zero cells arise when a cross-classification is never endorsed by both observers; biased marginal distributions occur when some categories are preferred differently between the observers. Simulations studied the distributions of the unweighted and weighted statistics for k = 4, under fixed proportions of diagonal agreement and different patterns off-diagonal, with various sample sizes, and under various zero cell count scenarios. Marginal distributions were first uniform and homogeneous, and then unbalanced asymmetric distributions. Results for unweighted kappa and B statistics were comparable to work of Muñoz and Bangdiwala, even with zero cells. A slight increased variation was observed as the sample size decreased. Weighted statistics did show greater variation as the number of zero cells increased, with weighted kappa increasing substantially more than weighted B. Under biased marginal distributions, weighted kappa with Cicchetti weights were higher than with squared weights. Both statistics for observer agreement behaved well under zero cells. The weighted B was less variable than the weighted kappa under similar circumstances and different weights. In general, B's performance and graphical interpretation make it preferable to kappa under the studied scenarios.
AB - Kappa and B assess agreement between two observers independently classifying N units into k categories. We study their behavior under zero cells in the contingency table and unbalanced asymmetric marginal distributions. Zero cells arise when a cross-classification is never endorsed by both observers; biased marginal distributions occur when some categories are preferred differently between the observers. Simulations studied the distributions of the unweighted and weighted statistics for k = 4, under fixed proportions of diagonal agreement and different patterns off-diagonal, with various sample sizes, and under various zero cell count scenarios. Marginal distributions were first uniform and homogeneous, and then unbalanced asymmetric distributions. Results for unweighted kappa and B statistics were comparable to work of Muñoz and Bangdiwala, even with zero cells. A slight increased variation was observed as the sample size decreased. Weighted statistics did show greater variation as the number of zero cells increased, with weighted kappa increasing substantially more than weighted B. Under biased marginal distributions, weighted kappa with Cicchetti weights were higher than with squared weights. Both statistics for observer agreement behaved well under zero cells. The weighted B was less variable than the weighted kappa under similar circumstances and different weights. In general, B's performance and graphical interpretation make it preferable to kappa under the studied scenarios.
KW - Bangdiwala's B
KW - Cohen's kappa
KW - Observer bias
KW - Zero cell
UR - http://www.scopus.com/inward/record.url?scp=41449119287&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=41449119287&partnerID=8YFLogxK
U2 - 10.1080/02664760701835052
DO - 10.1080/02664760701835052
M3 - Article
VL - 35
SP - 445
EP - 464
JO - Journal of Applied Statistics
JF - Journal of Applied Statistics
SN - 0266-4763
IS - 4
ER -