Inter-coder reliability is the most often used quantitative indicator of measurement quality in content studies.  Researchers in psychology, sociology, education, medicine, marketing and other social science disciplines also use reliability to evaluate the quality of diagnosis, tests and other assessments. While more than a dozen indices of inter-coder reliability are available, methodologists debated which indices are more suitable for general use (Zhao, Liu and Deng, in press).    

Zhao, Liu and Deng (in press) analyzed 22 indices of inter-coder reliability, including the first introduced -- percent agreement (a_o) and Benini’s β (1901), the most often “reinvented” -- Bennett et al’s S (1954), the most often used -- Scott’s π (1955) and Cohen’s κ (1960), the most strongly advocated-- Krippendorff’s α (1970), and the most recently introduced -- Gwet’s AC₁ (2008). Based on their analysis of the mathematics and the previously hidden assumptions, Zhao, Liu and Deng (in press) produced two liberal-conservative hierarchies for the 22 indices. The authors argued that researchers, reviewers and editors need to know, in general, which indices tend to generate higher scores and which tend to generate lower scores, so that they can interpret the numbers produced by these indices on a more informed basis.

This study is to extend Zhao, Liu and Deng’s (in press) work. While we agree that such hierarchies are useful, we argue that hierarchies based on mathematical analysis alone may not tell the whole story. We need alternative methodologies for cross verification. Accordingly, this study uses simulated data to build six more hierarchies. The resulted hierarchies show many similarities with Zhao, Liu and Deng’s (in press) results but also show some differences. In conclusion, these hierarchies together may be useful to workbench researchers who wish to better evaluate the inter-coder reliability indices of their instruments, and we provided software at (http://reliability.hkbu.edu.hk/) that helps to calculate these indices.