Compare Paired AI Evaluation Outcomes Before Calling a Change an Improvement

Two AI evaluation summaries can show the same accuracy while disagreeing on which cases were handled correctly. One candidate may fix several earlier errors and introduce equally many different errors. Another may produce exactly the same correct and incorrect outcomes as the first. Those situations share a headline percentage but raise different questions about regression, fallback work and the evidence needed for a change. Preserve the case pairings before asking an assistant to explain the comparison.

This guide uses six original fictional teaching cases to explain paired binary outcomes for two candidates against a fixed supplied reference. All outcomes and identifiers are deliberately authored. They are not observed model outputs, client evaluations or professional judgments. No model, statistical test or statsmodels function was executed. Independent local checks cover the supplied counts, identifier alignments and exact fractions. The examples demonstrate how to read a comparison, without establishing sampling uncertainty, business value or permission to release either candidate.

Start with the question the paired table can answer

A paired comparison asks what happened to the same eligible case under candidate A and candidate B. For this article, each eligible case has a supplied binary result for each candidate: match or mismatch against the same declared reference and rubric. Match means agreement with that supplied reference. It does not establish that the reference is correct in the world. The reference basis remains a separate part of evaluation design, alongside the case source and selection method.

The pairing carries information that two separate totals do not. If candidate A matches eighty of one hundred cases and candidate B matches eighty too, we know their marginal match counts. We do not yet know whether the same eighty cases matched. A table that retains both outcomes for each case can distinguish an unchanged set from a mixture of fixes and regressions. This is the practical reason to retain the joint outcomes rather than merely placing two percentages next to each other.

Use the same reference version when interpreting a candidate change. If the reference changes between runs, an apparent fix could come from relabeling rather than a changed candidate answer. Similarly, changed source material or changed acceptance criteria introduces another difference. These may be legitimate evaluation revisions, but they need their own explanation. Calling every difference a model improvement would collapse distinct changes into one claim that the count table cannot support.

The existing confusion-matrix guide explains one candidate's predictions against reference categories. The reviewer-agreement guide compares two supplied labeling judgments. This guide instead compares two candidates' match states on the same declared cases. Each view can be useful, but the axes mean different things. Write the axes in full whenever a table moves into a slide, spreadsheet or AI prompt so a familiar square of counts does not invite the wrong interpretation.

Read all four joint outcome cells

Declare rows as candidate A match or mismatch, and columns as candidate B match or mismatch. In that order, the four cells are both match, A matches while B mismatches, A mismatches while B matches, and both mismatch. This article calls the second cell a regression direction and the third a fix direction solely relative to the supplied reference and the change from A to B. Those words do not assign severity or business consequences to a case.

Fictional case A has counts six, two, two and zero in that order. There are ten eligible paired cases. Candidate A matches the six shared matches plus its two exclusive matches, giving eight of ten. Candidate B also matches eight of ten. Their accuracy difference on this authored population is zero percentage points. Yet four of ten cases change match state: two move in each direction. Equal accuracy has not preserved the set of matched cases.

Keep the shared cells visible. Both-match cases contribute to the marginal totals but do not distinguish which candidate matches more cases. Both-mismatch cases may be important substantive problems even though they also do not favor either candidate in the paired count difference. Excluding those cells from a presentation can make the remaining cases seem like the entire evaluation. Report the full eligible denominator first and then identify the discordant subset explicitly.

For these two match counts, subtracting A from B cancels the both-match cell. The difference equals fixes minus regressions. Dividing that net count by the full eligible paired denominator gives the difference in match fractions. Dividing by the discordant count answers another question: the balance among changed match states. Preserve the denominator in each sentence. A conditional percentage among changes should not be presented as a percentage of all evaluated cases.

Compare unchanged performance with balanced turnover

Fictional case B uses counts eight, zero, zero and two. Again there are ten paired cases, and each candidate matches eight. This time no case changes match state. The same eight cases match under both candidates and the same two mismatch under both. Cases A and B therefore have identical marginal accuracy fractions but different joint structures. The authored comparison supplies no real model behavior; it illustrates information lost when only the marginal fractions survive.

In case A, the changed-state fraction is four tenths, or two fifths. In case B it is zero tenths. The net change remains zero in both. A reader who asks only whether accuracy increased will miss the distinction. A reader investigating regressions needs the identifiers behind the two regression-direction cases in A, even if the two fix-direction cases offset them numerically. Offset in a count is not evidence that the consequences are equal.

Avoid describing zero net change as no change without naming the quantity. No net change in match count is accurate for both examples. No changed match states is accurate only for B. Even B does not prove identical text outputs: two different answers can both match a binary rubric, and two different wrong answers can both mismatch. The binary table retains a particular outcome comparison rather than every property of the underlying answers.

When a real report contains only two accuracy totals, do not reconstruct the missing joint cells by convenience. Many joint arrangements can be compatible with the same margins. Request the aligned case-level outcomes or a justified joint table. If those records are unavailable for the current comparison, state the gap. The absence of a paired table does not prove a regression happened, and it does not justify assuming there were none.

Separate net gains from gross fixes and regressions

Fictional case C changes the four counts to five, one, three and one. Candidate A matches six of ten; candidate B matches eight of ten. The difference is two of ten, or one fifth, which is twenty percentage points. Three authored cases move from mismatch to match and one moves from match to mismatch. Four cases change state overall. Those counts make the source of the net difference visible without assigning a real-world outcome to either candidate.

Among the four discordant cases, the fix-direction fraction is three quarters and the regression-direction fraction is one quarter. Among all ten eligible cases, the corresponding fractions are three tenths and one tenth. Both statements are arithmetically valid, but they answer different questions. A summary that says seventy-five percent improved without identifying the discordant denominator would be easy to misread as seven or eight of all ten cases improving.

Do not replace a percentage-point difference with a relative percentage change. Moving from six tenths to eight tenths adds two tenths of the full denominator. Relative to the original six-tenths fraction, that is a one-third increase. Neither arithmetic form measures additional revenue, time saved or production reliability. Use the form that fits the actual reporting question and show the original counts so a reader can reconstruct it without guessing which baseline was used.

The one regression-direction case deserves substantive inspection even though the net count is favorable to B. A binary match table has no severity scale and no indication of whether fallback can detect the mismatch. It also says nothing about latency, expense or user effort. Those are separate measurements and decisions. Keep them attached to actual evidence rather than inventing a benefit-cost conclusion from three fixes minus one regression.

Treat a test as a separate analytical decision

The official statsmodels McNemar documentation describes a two-by-two table interface and distinguishes an exact binomial option from a chi-square approximation, with a continuity-correction option for the latter. Its description identifies the discordant directions in the exact statistic and states a null hypothesis concerning equal marginal distributions. That limited context explains why a paired table can matter to a later statistical analysis. This article does not run either option or report a p-value.

Before requesting such an analysis, establish the actual study design and whether its assumptions fit the data. Cases linked by customer, document family or time may not supply independent evidence merely because each row has a different ID. Repeated tuning after viewing evaluation cases also changes the interpretation of a final comparison. The group-and-time split guide addresses those information-exposure questions. A correct arithmetic table cannot establish independence or untouched evaluation history.

Do not ask an assistant to choose a favorable test option after seeing results. If an inference is required, define the method, eligible population, missingness handling and comparison question with the responsible analyst before interpreting its output. This guide gives no universal minimum discordant count, significance threshold or sample-size recommendation. Those choices require more than the fictional counts supplied here and should remain distinguishable from descriptive arithmetic.

Likewise, a statistical result would not itself authorize a system change. A claim about marginal match probabilities is a different proposition from a claim that regressions are acceptable or operating safeguards work. Preserve the descriptive table even when a justified inferential analysis accompanies it. Readers should be able to identify what was counted, what was inferred and what remains a separate substantive decision rather than treating a test name as an overall approval stamp.

Align identifiers before comparing outcomes

Fictional case D supplies A records K1 match, K2 match, K3 mismatch and K4 mismatch. The B export arrives in order K3 mismatch, K4 mismatch, K1 match and K2 match. Under the explicitly supplied identifier mapping, every case has the same match state across candidates. The resulting joint counts are two, zero, zero and two. Both candidates match two of four cases and there are no discordant pairs.

Pairing those exports by position instead creates match/mismatch, match/mismatch, mismatch/match and mismatch/match. The four cells become zero, two, two and zero. Marginal match counts remain two each, while every positional pair appears discordant. The outcome values did not change; the pairing rule changed. This is a narrow check of the authored mapping, not certification that a real join or identity-resolution process is correct.

For an actual comparison, keep the case identifier, input version, reference version and candidate version together. Investigate duplicate identifiers and inconsistent versions before forming a joint table. A shared identifier can be insufficient when the underlying case content changed. Conversely, differently spelled identifiers may require an established mapping rather than automatic rejection or merging. The example supplies no automatic resolution procedure and recommends no normalization policy for real identities.

Ask for a reconciliation of the eligible case set before interpreting a regression list. Equal file lengths do not prove common membership. A sorted export may look aligned while silently omitting one case and adding another. The list-difference guide distinguishes presence, occurrence count and order; those checks support this task without replacing the substantive pairing decision. If the mapping remains unresolved, retain that state rather than naming an arbitrary positional mismatch a model regression.

Keep missing candidate outcomes outside a completed comparison

Fictional case E has four supplied IDs. E1 matches under both candidates. E2 mismatches under A and matches under B. E3 matches under A but has an unknown B result. E4 mismatches under both. Under the declared complete-pair rule, three cases are eligible. Their joint counts are one, zero, one and one. A matches one of three and B matches two of three on that complete subset.

The complete-pair fraction is three quarters of the supplied four records. It describes outcome availability under the stated rule, not semantic coverage, model quality or representativeness. The net difference on the complete subset is one third. The full four-case difference remains unresolved because E3 could retain its match or move into the regression direction. Do not silently treat its missing B result as mismatch, match or an intentional abstention.

Two hypothetical resolutions make the boundary clear. If E3 later matches under B, A matches two of four and B matches three, giving a one-quarter difference. If E3 mismatches under B, both match two of four, giving zero difference. These alternatives are arithmetic possibilities for the authored record, not observations or predictions. Neither may replace the unresolved full comparison while the B outcome is unavailable.

In an actual evaluation, record why an outcome is missing where known. A skipped case, an export failure and a defined abstention can require different treatment. If the task counts failure to return an answer as an outcome, specify that rule explicitly and apply it consistently. This example deliberately supplies no cause for the unknown result. Complete-pair reporting should not be presented as proof that exclusion is unbiased or that a full evaluation has finished.

Do not let a pooled total hide opposing group changes

Fictional case F contains Cedar counts two, zero, two and zero, and Maple counts two, two, zero and zero. Each group has four eligible cases. In Cedar, A matches two of four and B matches four of four, a one-half increase. In Maple, A matches four of four and B matches two of four, a one-half decrease. Pooling gives counts four, two, two and zero: both candidates match six of eight overall.

The pooled net difference is zero, but the group changes point in opposite directions. These invented group names have no demographic or client meaning. The example makes no fairness finding and does not identify the cause of a difference. It shows why a pooled description cannot be rewritten as improvement for every group, and why a group result needs its own denominator rather than being inferred from the overall fraction.

Retain group definitions before comparing them across evaluations. A case reassigned to another group or a changed inclusion rule can alter the apparent pattern. Also distinguish counts from evidence strength: four authored rows do not establish that any real group's performance is well characterized. If the actual task needs group-specific criteria, those criteria require substantive justification and a suitable design rather than thresholds chosen to flatter a particular candidate.

A practical comparison note can therefore report the pooled result, the group counts and the unresolved operating questions together. It should not average away a regression group simply because another group offsets it. Equally, it should not diagnose discrimination or causation from these tables alone. The useful next step is a targeted examination of actual cases and context, preserving uncertainty about explanations that the supplied counts do not establish.

Request an AI explanation with traceable arithmetic

Supply the aligned eligible rows, the reference and rubric versions, the orientation of the joint table and the handling of unknowns. Ask an assistant to return the four cells before any conclusion, then show both marginal match counts, gross fixes, gross regressions and net difference. Require every percentage to name its denominator. These are original suggested instructions for a review task, not a tested prompt or a promise that a model will follow them.

Check the proposed explanation against the rows before circulating it. Look specifically for swapped off-diagonal labels, a discordant denominator presented as the full population, or a zero net difference rewritten as identical outputs. Keep illustrative possibilities separate from observed outcomes. A fluent narrative can invent a cause, consequence or confidence claim even when its subtraction is correct. Retaining the count table gives a person a concrete basis for correcting that narrative.

Finish the note with the actual decision still required. A descriptive advantage may motivate more investigation without settling release readiness. Name missing reference evidence, unresolved case mappings or substantive regression questions in plain language. Preserve the original and corrected comparison when an input changes. The purpose is an understandable account of the supplied evidence that supports a responsible next step, with no invented professional review, operating authorization or business result.

Frequently asked questions

Can equal accuracy conceal regressions?

Yes. Case A has two regression-direction and two fix-direction cases while both candidates match eight of ten. Case B has the same marginal accuracy with no changed match states. These fictional examples demonstrate different joint structures; neither measures an actual model or establishes the consequences of a mismatch.

What is the difference between a fix count and a net gain?

A fix count includes cases where A mismatches and B matches under the supplied reference. Net gain subtracts the opposite direction. In case C, three fixes minus one regression gives two net matches across ten cases. The favorable net count does not establish that the one regression is acceptable.

Should the net difference use the discordant denominator?

The difference between the candidates' match fractions uses the full eligible paired denominator. A proportion among discordant cases answers a conditional question about changed states. Case C has three fixes among four changes, but three fixes among ten eligible cases. Always state which population the fraction describes.

Does this guide perform a McNemar test?

No. Official statsmodels documentation supplies limited context about a paired two-by-two test interface and its options. No statistical test, software scoring function, p-value or uncertainty interval was produced. The downloaded teaching cases support narrow literal, alignment and arithmetic checks only, without proving assumptions for inference.

Can I score two exports in their received order?

Only an established pairing rule can justify which records belong together. Case D's explicit fictional IDs give no discordance after alignment but complete discordance under positional pairing. Equal file lengths are insufficient. A real mapping needs the relevant identity and version evidence; this example certifies no real join.

Do missing results count as regressions?

That depends on the declared task rule. Case E uses complete pairs and keeps the full comparison unresolved. Missing is not silently converted to mismatch or intentional abstention. If a real evaluation defines no returned answer as a scored outcome, state that policy explicitly and keep it distinct from an unavailable export field.