Does this match SPSS?

Yes. The values follow Shrout and Fleiss (1979) and McGraw and Wong (1996), the same definitions used by SPSS RELIABILITY and the R psych package. SPSS's 'two-way mixed, absolute agreement' equals ICC(2,·) here.

What is a good ICC value?

Koo and Li (2016): below .50 poor, .50–.75 moderate, .75–.90 good, and above .90 excellent. Base the judgement on the 95% confidence interval as well as the point estimate.

Single measures or average measures?

Report single measures if one rater's score will be used in practice. Report average measures only if the final score is the mean of all raters.

INTRACLASS CORRELATION CALCULATOR

Intraclass correlation (ICC) calculator with 95% confidence intervals.

Measure inter-rater or test-retest reliability for numeric ratings. You get all six Shrout and Fleiss ICC forms with confidence intervals and F tests, help choosing the right one, and an APA sentence.

RATINGS

Rows = subjects, columns = raters

SHROUT & FLEISS

SCORES (HEADER ROW OPTIONAL)

For test-retest reliability, put each occasion (time 1, time 2) in its own column.

WHO ARE THE RATERS?

WHAT COUNTS AS AGREEMENT?

WILL YOU USE ONE RATER OR THE AVERAGE?

Intraclass correlation: which ICC to use, interpretation, and reporting

Which ICC should I report?

Different raters rate each subject → ICC(1)
Raters are a random sample and you need the same scores → ICC(2), absolute agreement
These are the only raters you care about → ICC(3); choose consistency if only the ranking matters
Use the ',1' single-rater form if one rater will be used in practice, the ',k' form if you will use the average
Test-retest reliability → usually ICC(2,1) absolute agreement

Formulas (two-way ANOVA mean squares)

ICC(2,1) = (MSR − MSE) ÷ [MSR + (k−1)MSE + k(MSC − MSE)/n] ICC(3,1) = (MSR − MSE) ÷ [MSR + (k−1)MSE]

MSR is the between-subjects mean square, MSC the between-raters mean square, MSE the residual, n the number of subjects and k the number of raters.

Worked example (Shrout & Fleiss, 1979)

Four judges rated six targets. ICC(3,1) = .71 (moderate) but ICC(2,1) = .29 (poor): the judges ranked targets similarly but used very different parts of the scale. That is why the choice between consistency and absolute agreement matters.

ICC, Cohen's kappa, or Cronbach's alpha?

Numeric ratings from two or more raters → ICC
Category ratings from two raters → Cohen's kappa
Internal consistency of questionnaire items → Cronbach's alpha (equivalent to ICC(3,k) consistency)

How to report it (APA style)

“Inter-rater reliability was assessed with a two-way random-effects, absolute-agreement, single-rater ICC. Reliability was poor, ICC(2,1) = .29, 95% CI [.02, .76].” Koo and Li (2016) recommend naming the model, type, and unit, and reporting the 95% CI.

References

Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86(2), 420–428.
McGraw, K. O., & Wong, S. P. (1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1(1), 30–46.
Koo, T. K., & Li, M. Y. (2016). A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of Chiropractic Medicine, 15(2), 155–163.

Intraclass Correlation (ICC) Calculator

Assess inter-rater and test-retest reliability for numeric ratings with all six Shrout and Fleiss ICC forms and a guided model choice.

Check agreement between raters scoring the same subjects

Measure test-retest reliability of a scale

Report the ICC model, type, unit, and 95% CI

KEY FUNCTIONS

All six Shrout and Fleiss ICC forms95% confidence intervals and F testsGuided choice of model, type, and unitKoo and Li (2016) interpretationAPA report sentence

Browse research statistics & survey tools →