Read Reviewer Agreement and Cohen Kappa Before Treating Labels as Truth

Two people can agree on most labels while still leaving important questions about the labeling task unanswered. They may have used different criteria, reviewed different versions of a case or selected one category almost every time. Agreement is useful evidence about a defined comparison, but it is not automatically evidence that either person identified the correct answer. Before using reviewer agreement to support an AI evaluation, preserve the cases, category meanings, pairing rule and missing-label treatment behind the number.

This guide explains unweighted Cohen's kappa alongside the ordinary agreement fraction using six original fictional examples. All tables, row identifiers and labels are deliberately authored. They do not represent professional reviewers, actual annotations, client records or observed AI outputs. No scikit-learn scoring function or model was run. Local exact-fraction checks cover the supplied arithmetic and pairings only. The purpose is to make a proposed evaluation's labeling evidence understandable without turning a statistic into a certification of truth, reviewer competence or deployment readiness.

Define what the two label columns represent

Start by naming the task and its categories. A comparison might concern whether a draft preserves a stated date, whether a source supports a sentence, or how an example fits an explicitly defined classification. Those are different judgments. A broad label such as acceptable can conceal several requirements. If the reviewers apply different meanings, a disagreement count cannot tell you which requirement caused the difference. Establish the actual rubric and retain the source context needed to apply it.

The teaching examples use opaque category labels A and B. They intentionally assign no approval, severity or business meaning to either category. An A label is not a correct answer, an authorization or a favorable outcome. This keeps the arithmetic separate from a substantive judgment that the fictional data does not support. For a real comparison, the category definitions must come from the actual task and its responsible owners, rather than from a scoring function's expectations about the label strings.

Also state whether the columns are independent first-pass judgments, labels produced after discussion, or a first label compared with an adjudicated reference. These are different evidence roles. If the people already discussed the case and agreed on an answer, later matching labels should not be described as untouched independent judgments. Conversely, initial disagreement can be useful information for clarifying the rubric without proving that either reviewer is careless or that the task is impossible.

The scikit-learn documentation defines Cohen's kappa for two annotators and describes its symmetry: exchanging their columns does not change the statistic. That mathematical symmetry does not make their operating roles interchangeable. One column might be a provisional label while another has a documented reference basis. Keep that role information outside the score so a symmetric calculation does not erase an asymmetric evidence relationship.

Read the paired-label table before calculating a score

For two categories, use a two-by-two count table with the first labeler's categories as rows and the second labeler's categories as columns. The top-left count represents A/A pairs, the top-right A/B pairs, the bottom-left B/A pairs and the bottom-right B/B pairs. State that orientation explicitly. A table without labels can be mistaken for a model confusion matrix in which one side is established truth, even when both sides are simply supplied judgments.

Fictional case A uses counts four, one, one and four in that order. There are ten paired cases. Eight lie on the matching diagonal, and two are disagreements. The observed agreement fraction is eight tenths, or four fifths. Each labeler assigns A to five cases and B to five. These totals are properties of the authored table; they provide no observations about an actual review team or prevalence in a business population.

The table also preserves information that the agreement fraction alone omits. It shows one disagreement in each direction and balanced marginal label counts. A summary saying eighty percent agree would conceal that structure. Retaining the table allows a reader to check the denominator and see how the categories were used, without needing to infer the underlying counts from a rounded percentage or from the author's interpretation of whether the number seems high.

Do not silently discard an off-diagonal direction because one category is considered more important in the real task. If the task needs different consequences for different disagreements, define that separate analysis and its rationale. This article calculates unweighted nominal agreement only. It does not recommend severity weights, ordered-category distances or a release threshold, and it does not convert either off-diagonal count into a monetary or operating consequence.

Separate observed agreement from the marginal-based comparison

The documented kappa formula is the difference between observed agreement and a marginal-based expected agreement, divided by one minus that expected agreement. For these unweighted examples, compute the expected component by multiplying the two labelers' A proportions, doing the same for B, and adding those products. This is a mathematical reference constructed from the supplied marginal proportions. It is not a direct observation that the people guessed, a measurement of their motives or proof that their decisions were random.

In case A, each labeler uses A on one half of the cases and B on one half. The expected component is one half times one half plus one half times one half, giving one half. Observed agreement is four fifths. Subtracting one half gives three tenths; dividing by the remaining one half gives kappa three fifths. The value is exact under the supplied counts and formula, with no model execution or uncertainty interval attached.

Keep both quantities in a report when they matter. Observed agreement answers how many paired labels match in the included set. Kappa compares that match fraction with the particular marginal-based reference. Neither quantity says what the original examples mean or whether matching judgments are correct. A clear explanation names the input pairs and method rather than using a value alone as a claim that the rubric has been validated.

The denominator deserves inspection as well. When the expected component is one, the displayed formula divides by zero. That is a distinct mathematical state, not automatically a zero or perfect kappa. A software implementation can have a documented way of representing an undefined result, but a chosen display value should not conceal why the expression was undefined. This guide retains that distinction explicitly in its degenerate example.

Compare equal raw agreement with different category use

Fictional case B changes the four counts to seven, one, one and one. The total remains ten, and eight pairs still match. Observed agreement is again four fifths. Each labeler now assigns A to eight cases and B to two. The raw match fraction is the same as in case A, but the marginal-based expected component changes because the label distributions differ. There is no actual change in reviewer behavior measured here; both tables were authored to illustrate the comparison.

The expected component in case B is eight tenths squared plus two tenths squared, or sixty-eight hundredths, equal to seventeen twenty-fifths. The numerator is four fifths minus seventeen twenty-fifths, which is three twenty-fifths. The denominator is eight twenty-fifths. Their ratio is three eighths. Thus the two authored tables have the same observed agreement while producing kappa three fifths and three eighths respectively.

This comparison is a reason to inspect category counts before ranking review processes by a single statistic. The examples do not establish that the second process is worse, that common categories should be reduced or that a particular value is acceptable. A real comparison needs the tasks, populations and category meanings. Changing the case mix can change the marginal proportions, so a statistic from one set should not be treated as a context-free property of the reviewers.

Do not rebalance a real evaluation merely to make its kappa more attractive. A deliberately balanced teaching set can help explain edge cases, while an operational sample may have another purpose. Report the selection method and its consequences for the denominator. If several sets answer different questions, describe them separately. An apparently improved score obtained by changing the included examples is not automatically an improvement in the labeling procedure.

Treat systematic disagreement as a question about the rubric

Fictional case C uses counts zero, four, four and zero. None of the eight paired labels match. Each column still contains four A labels and four B labels, so the expected component is one half. The formula gives zero minus one half, divided by one half, or negative one. That exact result describes the authored opposite pairings under the specified nominal formula. It does not show that either labeler is wrong on every case, because no truth reference has been supplied.

A real pattern of opposite labels could lead to several different questions. Did the reviewers receive reversed category definitions? Were they looking at different versions of a source? Was one interpreting a yes-or-no instruction differently? The count table alone cannot diagnose the cause. Inspect representative disagreements with their original evidence and instructions before assigning blame or changing labels. A numerical result can direct a useful investigation while leaving its explanation unresolved.

If discussion resolves a definition problem, retain both the original independent labels and the later agreed interpretation where appropriate for the actual task. They answer different questions. Replacing the first-pass table with a post-discussion table and calling it independent agreement would lose the sequence that explains the change. This article performs no actual adjudication and identifies no real reviewer, but the evidence-role distinction is central to an interpretable evaluation.

A matching pair can also share a misunderstanding. Agreement statistics cannot detect that merely from the labels. Where correctness matters, the task needs a justified source comparison or reference process in addition to agreement. The related AI pilot and classification guides address different parts of that evaluation. Do not use the fact that two people agree as a substitute for establishing what supports the answer in the first place.

Keep an undefined result distinct from a failed review

Fictional case D contains four A/A pairs and no other pairs. Observed agreement is one: all four supplied labels match. Both marginal distributions put all their mass on A, so the expected component is also one. The numerator and denominator of the kappa expression are both zero. The fraction is undefined. Reporting perfect raw agreement is consistent with the supplied table; reporting a mathematically defined perfect kappa from this expression would not be.

This is not a reason to relabel one case artificially or add an invented disagreement. It is a reason to state the limitation. The included examples contain no variation in assigned category, so the marginal-adjusted expression cannot provide the same kind of comparison as the earlier cases. The example supplies no inference about how the labelers would treat B cases or whether enough examples were selected for an actual study.

Software behavior and interpretation should be separated. The current documented scikit-learn interface includes handling for undefined results. This article does not execute that interface, test its warnings or claim a particular installed version's behavior. Its exact-fraction teaching calculation deliberately records undefined when the denominator is zero. If a real report uses a replacement value, name that convention and retain the undefined condition so the display is not mistaken for an observed score.

The same discipline applies to an empty eligible set. No paired labels means no observed agreement denominator. This guide does not add an empty-set case to the six supplied tables, but the reporting question is straightforward: identify what was available to compare before claiming a result. A blank output, an undefined calculation and a measured value are different evidence states. Keeping them separate makes the next data or rubric question clearer.

Align the cases before aligning the label arrays

Fictional case E supplies first-labeler records K1/A, K2/A, K3/B and K4/B. The second-labeler records are ordered K3/B, K4/B, K1/A and K2/A. Matching by the explicitly supplied case identifiers produces four matching label pairs. The observed agreement and defined kappa are both one because the aligned marginals are balanced. That result concerns the declared fictional identifier mapping; it is not a finding that a real-world identity join is correct.

Pairing the arrays by position instead produces A/B, A/B, B/A and B/A. All four mismatch, giving observed agreement zero and kappa negative one under the same balanced marginals. The individual labels did not change. The pairing rule changed which examples were compared. This is why an agreement calculation needs aligned case identity rather than a belief that two exports happen to be in the same order.

In an actual handoff, preserve the relevant case identifier and version with each label. Check duplicate identifiers, absent cases and inconsistent versions before scoring. This article offers no automatic identity-resolution procedure and does not advise collapsing duplicates by convenience. If the mapping is unresolved, retain that problem rather than interpreting an arbitrary pairing as a disagreement between people. A precise score cannot repair an unsupported case alignment.

The existing list-difference owner distinguishes membership, occurrences and order, and the classification confusion-matrix guide retains paired references and predictions. Those are related but separate questions. Here both label columns can be provisional judgments, so neither becomes established truth simply because one is placed first. The aligned comparison says which supplied labels match; any correctness interpretation requires its own evidence basis.

Preserve missing labels and the eligible denominator

Fictional case F has three supplied case IDs. K1 has A/A, K2 has A/B and K3 has B with the second label missing. Under the explicitly declared complete-pair rule, two of the three cases are eligible for the agreement calculation. One of those two pairs matches. The paired observed agreement is one half. The available-pair fraction is two thirds, which measures supplied-label availability rather than semantic coverage or annotation quality.

For the two eligible pairs, the first labeler uses A twice; the second uses A once and B once. The expected component is one half, and kappa is zero. That is a result for the two known pairs only. The full three-case agreement remains unresolved. Replacing the missing label with A or B would create an additional assumption and could change the match fraction, so the example does not silently perform either substitution.

A missing label can arise for many reasons in an actual process, including an incomplete export, an intentional abstention or a case not yet reviewed. The fictional fixture does not assign a cause. If abstention is a meaningful task category, define it as such; if the field is unavailable, keep it unavailable. Treating those states as identical would change the population and the meaning of the calculated statistic without making that change visible.

Before using agreement in an AI evaluation, connect the label note to the broader dataset design. Record the rubric version, pairing rule, included cases, missingness treatment and whether judgments were independent. The contextual Dr Connor evaluation-dataset guide addresses that broader design, while this page explains the narrower statistic and its prerequisites. Neither supplies a professional review or authorization to operate a system. The useful outcome is an inspectable comparison whose limits remain attached to its number.

Retain counts when displaying rounded scores

Keep the count table alongside any rounded display. Three eighths is exactly 0.375, while three fifths is exactly 0.6 in the authored calculations. Rounding both to a coarse verbal label would hide the comparison the examples were designed to explain. Conversely, printing many decimal places does not add information about sampling uncertainty or category meaning. Precision in arithmetic is different from precision in an inference about a population.

If the included case set changes, recompute the paired table and its marginal proportions together. Do not update the observed agreement numerator while leaving the old expected component in place. Adding a pair changes the total and may change both label distributions, so kappa generally cannot be updated by averaging the old value with a score for the new case. This guide supplies no incremental scoring utility. Retaining the table makes the full recomputation inspectable and keeps a corrected display connected to the exact included cases.

Frequently asked questions

Does reviewer agreement establish that the labels are correct?

No. Two supplied judgments can match without a truth reference. Agreement describes their relationship on the included cases. Correctness needs an independent basis appropriate to the task, such as the actual source and a justified reference process; none is supplied by these fictional examples.

Why can two tables with eighty percent agreement have different kappa?

The marginal label proportions can differ. Cases A and B both have eight matching pairs out of ten, but their expected components are one half and seventeen twenty-fifths. Their resulting kappas are three fifths and three eighths. These authored values are not rankings of actual review teams.

Is perfect raw agreement always a defined kappa of one?

No. Case D assigns A to every supplied case in both columns. Observed agreement and the marginal-based expected component are both one, leaving a zero denominator. The kappa expression is undefined even though all four labels match. The guide does not substitute a convenient display value.

Can I compare label arrays in the order I received them?

Only a justified pairing rule can establish which cases belong together. Case E's supplied identifiers produce complete agreement after alignment and complete disagreement under naive positional pairing. The example validates its explicit fictional mapping, not a real identity-resolution procedure or arbitrary export ordering.

Should a missing second label be counted as a disagreement?

The answer depends on the declared task and method. Case F explicitly uses complete pairs and reports two-thirds label availability separately. It does not treat missing as A, B or an error. A real evaluation should retain the reason and rule where known rather than silently changing the denominator.

Do the downloaded tables evaluate a model or professional reviewers?

No. They are original fictional nominal-label exercises checked with exact fractions and supplied identifiers. No actual model, client records, scikit-learn scoring function, reviewer performance, confidence interval, professional assessment or operating approval is represented by the calculations.