What is a good Cohen's kappa?

Landis and Koch (1977) call 0.61–0.80 substantial and above 0.80 almost perfect. Many health and education studies expect at least 0.60; McHugh (2012) recommends 0.80 or higher for clinical research.

Can I use Cohen's kappa with three raters?

No. Cohen's kappa is for exactly two raters. With three or more raters, use Fleiss' kappa.

Why is my kappa low when agreement is high?

When one category is very common, chance agreement is high, so kappa drops even with high percent agreement. Report both values and explain the category balance.

COHEN'S KAPPA CALCULATOR

Cohen's kappa calculator for inter-rater agreement.

Measure how well two raters or coders agree beyond chance. Enter the agreement table and get kappa, its 95% confidence interval, a p value, the Landis and Koch interpretation, and weighted kappa for ordered categories.

AGREEMENT TABLE

Rows = rater 1, columns = rater 2

BROWSER-ONLY

Each cell is the number of items rater 1 put in the row category and rater 2 put in the column category. The diagonal holds the agreements. A header row of category names is optional.

COUNTS (SQUARE TABLE)

Use weights only when categories are ordered (for example Low, Medium, High). Quadratic weighted kappa is often reported alongside the intraclass correlation.

Cohen's kappa: formula, interpretation, and reporting

Formula

κ = (pₒ − pₑ) ÷ (1 − pₑ)

pₒ is the proportion of items the raters agreed on (the diagonal) and pₑ is the agreement expected by chance from each rater's category totals. κ = 1 is perfect agreement; κ = 0 is no better than chance.

Worked example

Two coders rate 50 interview excerpts as relevant or not. They agree on 35 (pₒ = .70). Coder 1 marks 25 as relevant and coder 2 marks 30, so pₑ = .5 × .6 + .5 × .4 = .50. κ = (.70 − .50) ÷ (1 − .50) = .40: fair agreement.

Cohen's kappa interpretation (Landis & Koch, 1977)

< 0 — Poor (less than chance)
0.01–0.20 — Slight
0.21–0.40 — Fair
0.41–0.60 — Moderate
0.61–0.80 — Substantial
0.81–1.00 — Almost perfect

Kappa, weighted kappa, or ICC?

Two raters, unordered categories → Cohen's kappa
Two raters, ordered categories → weighted kappa (linear or quadratic)
More than two raters, categories → Fleiss' kappa
Numeric ratings or scores → intraclass correlation (ICC), see the ICC calculator

How to report it (APA style)

“Two coders independently rated all 50 excerpts. Agreement was 70%, and Cohen's kappa indicated fair agreement, κ = .40, 95% CI [.15, .65], p = .004.” Report percent agreement, κ, the confidence interval, and the number of items.

References

Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37–46.
Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159–174.
Fleiss, J. L., Cohen, J., & Everitt, B. S. (1969). Large sample standard errors of kappa and weighted kappa. Psychological Bulletin, 72(5), 323–327.
McHugh, M. L. (2012). Interrater reliability: The kappa statistic. Biochemia Medica, 22(3), 276–282.

Cohen's Kappa Calculator

Measure agreement between two raters or coders beyond chance, with confidence intervals and weighted kappa for ordered categories.

Check intercoder reliability in qualitative coding

Measure agreement between two diagnosticians or graders

Report kappa with a confidence interval in a thesis

KEY FUNCTIONS

Unweighted, linear, and quadratic weighted kappa95% confidence interval and p valueObserved and chance agreementLandis and Koch interpretationAPA report sentence

Browse research statistics & survey tools →