How to Interpret Cohen's Kappa for Inter-Rater Agreement

Cohen's kappa tells you how much two raters agree on categorical judgments after accounting for the agreement their category choices would produce by chance. Interpret it by checking that kappa fits your design, reading its value alongside raw agreement and the category table, and examining uncertainty and category imbalance. A value such as 0.56 is not a self-explanatory grade: its meaning depends on the decision, the data, and the consequences of disagreement.

This guide is for students reading or conducting research with two people who classify the same items—for example, coding interview excerpts as supportive or critical, screening abstracts as include or exclude, or rating observations into named categories. You will learn what kappa measures, calculate a worked example, identify misleading interpretations, and report the result clearly.

Key takeaways

• Use ordinary Cohen's kappa for exactly two raters assigning the same items to nominal categories.
• Report the confusion matrix and observed percent agreement with kappa; each reveals something the other hides.
• Use weighted kappa for ordered categories when near and far disagreements should count differently.
• Do not treat a verbal label such as “moderate” as a universal pass mark.
• Check category prevalence, rater bias, sample size, and a confidence interval before drawing conclusions.

What Cohen's kappa actually measures

Cohen's kappa is a chance-corrected measure of agreement for categorical ratings. Jacob Cohen introduced it because percent agreement alone counts matches that can occur simply because both raters often choose the same category. The statistic compares observed agreement, written as Po, with expected agreement under independent category choices, written as Pe. The definition and its original purpose are described in Cohen's 1960 paper.

The formula is κ = (Po − Pe) ÷ (1 − Pe). The numerator is agreement beyond chance; the denominator is the maximum possible agreement beyond chance. Kappa equals 1 when agreement is perfect, 0 when observed agreement equals chance-expected agreement, and can be negative when raters agree less often than the model expects.

Kappa measures agreement, not validity or accuracy. Two coders can consistently apply the same mistaken rule and obtain high kappa. Conversely, a low value can indicate ambiguous definitions, inadequate training, a difficult judgment, skewed category frequencies, or several of these at once. The peer-reviewed overview by McHugh explains why inter-rater reliability matters to the quality of collected data.

First check whether kappa fits your design

Before interpreting a number, verify the design. Ordinary Cohen's kappa is appropriate when two raters independently classify the same set of items and the outcome is nominal: categories differ in kind but do not have a meaningful rank. Examples include relevant/not relevant, cat/dog/other, or complaint/praise/request.

Choose a different statistic when the design changes

• Ordered categories: Use weighted kappa when categories have a natural order, such as poor/fair/good/excellent. You must state the weighting scheme because linear and quadratic weights penalize disagreements differently.
• More than two raters: Consider a multi-rater statistic such as Fleiss' kappa or Krippendorff's alpha, depending on the design and missingness.
• Continuous measurements: Use a method designed for continuous agreement, often an intraclass correlation coefficient or a Bland–Altman analysis, rather than forcing measurements into arbitrary categories.
• Accuracy against a gold standard: Use accuracy, sensitivity, specificity, or another validation measure. Agreement between peers is a different question.

Wongpakaran and colleagues emphasize that ordinary Cohen's kappa suits two raters and nominal categories, while weighted kappa is designed for ordered categories. That choice is part of interpretation, not a minor software setting.

Worked example: calculate Cohen's kappa step by step

Suppose two students independently screen 50 article abstracts for a literature review. Each assigns “include” or “exclude.” Their decisions form this 2 × 2 table: they both include 22 abstracts, both exclude 17, Rater A alone includes 8, and Rater B alone includes 3.

Step 1: Calculate observed agreement

The raters agree on 22 + 17 = 39 abstracts. Therefore Po = 39 ÷ 50 = 0.78, or 78% observed agreement.

Step 2: Calculate agreement expected by chance

Rater A includes 30 of 50 abstracts and excludes 20. Rater B includes 25 and excludes 25. Expected include agreement is (30 ÷ 50) × (25 ÷ 50) = 0.30. Expected exclude agreement is (20 ÷ 50) × (25 ÷ 50) = 0.20. Add them: Pe = 0.30 + 0.20 = 0.50.

Step 3: Calculate kappa

Substitute the values: κ = (0.78 − 0.50) ÷ (1 − 0.50) = 0.28 ÷ 0.50 = 0.56. The raters achieved 56% of the agreement beyond chance that was still available.

Step 4: Interpret the result in context

A common benchmark would call 0.56 “moderate,” but that label is only a starting point. For an exploratory class exercise, the result may show a usable codebook that still needs refinement. For a high-stakes clinical classification, the same value could be inadequate. Review the 11 disagreements, determine which rule caused them, revise the codebook, and test a fresh sample rather than merely celebrating the label.

💡 Practical note: Keep adjudication separate from reliability calculation. Compute kappa from the independent first-pass ratings. Resolve disagreements afterward to produce the final dataset.

How to read a reported kappa in five checks

1. Confirm the unit and categories. What did each rater classify, and were categories nominal or ordered? A statistic cannot rescue vague or overlapping category definitions.
2. Read the full agreement table. Look beyond the diagonal matches. The off-diagonal cells show the direction and type of disagreement—for example, whether one rater includes far more abstracts than the other.
3. Compare kappa with percent agreement. A large gap can occur when one category dominates. Report both so readers can see the observed consistency and the chance correction.
4. Check uncertainty. A 95% confidence interval shows how imprecisely kappa is estimated. A point estimate of 0.56 with a wide interval supports less certainty than the same estimate from a much larger sample.
5. Judge fitness for purpose. Ask how disagreement affects the research outcome. Reliability requirements should be stricter when classification errors can change treatment, safety, or the primary conclusion.

Penn State STAT 509 presents kappa as observed agreement beyond agreement expected under independence and notes that negative values can occur. Martin Bland's University of York textbook excerpt demonstrates the calculation and shows why confidence intervals and category frequencies matter.

Four interpretation traps to avoid

Trap 1: Treating benchmark labels as universal standards

Ranges such as slight, fair, moderate, substantial, and almost perfect are conventions, not natural laws. Different fields face different costs of disagreement. State any benchmark you use, justify it, and interpret the actual estimate and interval in context.

Trap 2: Ignoring the prevalence problem

When almost every item belongs to one category, two raters may have high raw agreement while kappa remains unexpectedly low. The expected-agreement term becomes large because both raters frequently select the dominant category. Do not “fix” this by hiding kappa; show the category counts, percent agreement, and kappa, then explain the imbalance.

Trap 3: Calling kappa accuracy

Kappa does not reveal which rater is correct. If you have a defensible reference standard, compare each rater with that standard using validity measures. If no reference exists, describe the statistic as inter-rater agreement and avoid claims about correctness.

Trap 4: Testing only whether kappa exceeds zero

A statistically significant result can still be too small for the study purpose. The useful question is not merely whether agreement is better than chance, but whether the estimate is sufficiently high and precise for the decisions based on the ratings.

A better workflow for student research projects

1. Write operational definitions. Give each category an inclusion rule, an exclusion rule, and at least one borderline example.
2. Train on shared examples. Discuss why examples fit a category, then revise ambiguous wording before formal testing.
3. Pilot independently. Have both raters classify the same pilot set without seeing each other’s choices.
4. Inspect disagreements. Identify whether errors come from unclear categories, inconsistent units, missing context, or simple slips.
5. Revise and retest. After changing the codebook, use a fresh set of items. Reusing memorized cases can inflate apparent improvement.
6. Freeze the protocol. Record the final codebook version, statistic, weighting if any, sample size, software, point estimate, confidence interval, and percent agreement.

A simple evidence log can prevent definitions and decisions from drifting. In Snitchnotes, you can keep the codebook beside source notes, turn borderline examples into practice questions, and review why each example belongs to its category before the next pilot round.

Reporting template

Two raters independently classified 50 abstracts as include or exclude. Observed agreement was 78%. Cohen’s kappa was 0.56 (95% CI: [insert software output]), indicating agreement beyond chance that was not yet sufficient for our prespecified threshold of [insert justified criterion]. Disagreements were resolved after the reliability analysis.

Adapt the template rather than copying it blindly. Name the rated items, categories, number of raters, independence procedure, kappa variant, weighting scheme when relevant, number of observations, raw agreement, point estimate, confidence interval, and decision rule. If one category is rare, report category totals or the full table.

Frequently Asked Questions

What is a good Cohen's kappa value?

There is no universal good value. Common labels can help describe a result, but adequacy depends on the field, consequences of disagreement, category prevalence, and precision of the estimate. Define an acceptable threshold before analysis when possible, justify it, and report raw agreement and a confidence interval alongside kappa.

Can Cohen's kappa be negative?

Yes. A negative kappa means observed agreement is lower than the agreement expected from the raters’ marginal category frequencies. It can signal systematic disagreement, reversed rules, data coding errors, or a very unusual sample. Inspect the agreement table and coding procedure before attaching a substantive explanation.

Why is kappa low when percent agreement is high?

This often happens when one category is very common. Because both raters frequently choose that category, the model expects substantial agreement by chance, leaving less agreement beyond chance. Report the category counts, observed agreement, and kappa together so readers can see the imbalance rather than relying on one number.

When should I use weighted kappa?

Use weighted kappa for two raters and ordered categories when the distance between disagreements matters. Confusing “excellent” with “good” may deserve a smaller penalty than confusing “excellent” with “poor.” State whether weights are linear, quadratic, or otherwise defined because different choices can produce different results.

Is Cohen's kappa the same as correlation?

No. Correlation measures association, not exact agreement. Two raters can be perfectly correlated while one consistently assigns higher scores. For nominal categories, Cohen’s kappa evaluates matches beyond chance; for continuous measurements, choose an agreement method suited to the scale and design.

Conclusion

To interpret Cohen's kappa well, start with design fit, then read the estimate beside observed agreement, the full category table, and its confidence interval. Treat benchmark labels as descriptive conventions, not verdicts. If the result is inadequate, improve definitions, train with borderline cases, and retest independently on new items.

The practical goal is dependable data, not a flattering coefficient. Use Snitchnotes to keep definitions, examples, and revision questions connected, then report your reliability procedure transparently so readers can judge the evidence for themselves.

Sources

• Cohen, J. (1960). A coefficient of agreement for nominal scales.
• McHugh, M. L. (2012). Interrater reliability: the kappa statistic.
• Wongpakaran et al. (2023). Kappa statistic considerations in evaluating inter-rater reliability between two raters.
• Penn State STAT 509: Cohen's Kappa Statistic for Measuring Agreement.
• Bland, M. Assessing agreement using Cohen's kappa, University of York.