Interrater Agreement Index

Klein, K. J., Conn, A.B., Smith, D.B. and Sorra, J. S. (2001). Everyone agrees? A study of the coherence between groups in the perception of the work environment by staff. J. Appl. Psychol.

86, 3-16 doi: 10.1037/0021-9010.86.1.3 The advantages of using the Swg as an IRA index are that it is a common measure of variation and its interpretation is not complicated by the use of multi-item scales or nonlinear functions [see rwg(j)]. However, the Swg did not always appreciate a wide application. It cannot be explicitly compared to random response distributions, and that could be interesting. It also tends to increase with the size of the response options at scale, which means that comparisons on scales are not possible. Finally, it will also tend to decrease as the sample size increases; Therefore, it is not independent of sample size. Interrater Agreement (IRA) Applications Statistics for Likert scales are abundant in research and practice. The article also contains a quick reference table and three digits to help readers understand how ERI statistics differ and how ERI interpretations are highly dependent on the statistics used. A brief review of recommended practices with statistical standards and cut-off practices is presented and conclusions are proposed in the light of the current literature. Later extensions of the approach included versions that could handle “partial credits” and ordinal scales. [7] These extensions converge with the family of intraclassical correlations (CIC), so there is a conceptually related possibility of estimating reliability for each level of measurement, from nominal cappa to interval (ICC or ordinal kappa) through interval (ICC or ordinal kappa) and ratio (CIC).