Read Mean Reciprocal Rank Without Losing the First Relevant Result

Mean reciprocal rank summarizes where the first relevant result appears across a set of queries. That is useful when the practical question concerns reaching an initial useful item. It is a narrow question. The measure does not reveal how many other relevant items were retrieved, whether the first relevant passage is sufficient for an answer or whether an AI assistant used it correctly. Read the first-hit evidence before turning the summary into a statement about an entire retrieval workflow.

This guide builds an interpretation workflow using original fictional rankings, manually supplied binary judgments and exact arithmetic. No retrieval engine, embedding model, language model, external scoring library or client system was run. The examples establish only what follows from their declared lists and conventions. Relevance itself remains a supplied judgment, and none of the calculations validates a source, measures a real user's experience or grants permission to use information.

Define what counts as the first relevant item

Begin with one query and one ordered list. Name the retrieval unit, such as a versioned paragraph rather than an entire document. Mark which units are relevant under the evaluation question. The first relevant rank is the one-based position of the earliest relevant item in that list. Rank one therefore means the first position, while an array index of one could mean the second position in a programming language. Preserve that distinction in exports.

A relevance label needs a task definition. A paragraph about general request handling may be relevant to an overview query but insufficient for a query about an exception. A source can address the topic without resolving the specific question. Before scoring, state whether the binary judgment means topical usefulness, direct support for the requested fact or another reviewed criterion. Do not let the metric name supply an unstated interpretation.

For a fictional query about Workshop Note E4, a result from E3 might look similar and still belong to the wrong version. If version eligibility is part of the task, apply that rule consistently before calling the item relevant. Keep a separate record of excluded candidates when the evaluation needs to explain filtering. An item removed before ranking does not occupy a rank position in the eligible list unless the report explicitly defines otherwise.

The official TorchMetrics retrieval MRR interface provides primary documentation showing query identifiers, binary targets, top-k handling and configurable empty-target and aggregation behavior. These options are implementation context rather than a universal reporting contract. This guide calculates its own small authored examples directly and does not execute TorchMetrics, claim complete library conformance or adopt its example outputs as observed findings.

Take the reciprocal before taking the mean

Under the convention used here, a query whose first relevant item is at rank r receives reciprocal rank one divided by r. First place gives one, second place gives one half, third place gives one third and fourth place gives one fourth. A query with no relevant item inside an explicitly declared cutoff receives zero in these teaching examples. That zero is a chosen scoring rule for the bounded list, not proof that the collection contains no relevant item.

Mean reciprocal rank is the arithmetic mean of those per-query reciprocal values under the chosen query weighting. The reciprocal transformation happens before averaging. This order matters because taking one divided by the average rank generally gives a different number. A report that stores only an average rank cannot reconstruct mean reciprocal rank without the original per-query information.

Keep the word mean attached to a population. It could mean one equal contribution from each authored query, an average over repeated query occurrences or a weighted average chosen for a particular workload. Those are different objects. If a dashboard says MRR without naming its query set and weighting, ask for the row-level ranks and inclusion rule before interpreting a small change as an improvement.

The reciprocal scale also makes position changes unequal. Moving a first relevant item from rank two to rank one changes reciprocal rank by one half. Moving it from rank four to rank three changes the value by one twelfth. Both movements are one position, but their score differences are different. That arithmetic does not measure the corresponding difference in human effort or business value.

Case A: mean reciprocal rank is not reciprocal mean rank

The first fictional group contains three queries with first relevant ranks one, two and four. Their reciprocal ranks are one, one half and one fourth. The sum is seven fourths, and the equal-query mean is seven twelfths. This is approximately 0.5833 when displayed to four decimal places, but the exact fraction remains the clearer audit value for this small authored table.

The ordinary average of the three ranks is seven thirds. Taking its reciprocal gives three sevenths, approximately 0.4286. That is not seven twelfths. Nothing about the discrepancy suggests a rounding issue or a conflicting relevance judgment. The calculations apply different operations in different orders to the same inputs, so they answer different numerical questions.

For a practical reporting review, retain a column for each query's first relevant rank and another for its reciprocal value. Sum the reciprocal column and divide by the included query count. A reader can then reproduce the aggregate and inspect unusually influential rows. A formula applied to an average-rank cell would lose the information needed for that check.

Do not describe seven twelfths as the fraction of queries answered correctly. In this group every query has a relevant item somewhere in its supplied ranking, but their reciprocal values differ by position. A successful-query fraction under an at-least-one-hit rule would be a separate count with a separate meaning. Neither summary establishes whether an assistant produced a supported final answer.

Case B: identical first hits can hide different evidence coverage

The second group supplies two rankings of four distinct identifiers. In ranking S, only the second position is relevant. In ranking T, the second, third and fourth positions are relevant. Both first relevant ranks are two, so both reciprocal ranks are one half. Mean reciprocal rank cannot distinguish the additional later relevant items when the first relevant position stays unchanged.

This is not a defect in the arithmetic. The measure was defined around the first relevant item. If the task needs several pieces of evidence, such as a rule, an exception and a definition, use a separate inspection of whether those pieces reach the handoff. A first relevant paragraph could be useful while leaving the answer incomplete because the required qualification remains absent.

The authored rankings deliberately make no claims about which set would produce a better AI response. More relevant items could help provide context, but their content, compatibility and use are not specified. The only established comparison is that the first-hit score is equal while the count of relevant supplied items differs. Keep that bounded statement when explaining the worksheet to a colleague.

A useful review combines the first-hit trace with a coverage trace. The first records the earliest relevant identifier and its position. The second records which independently required evidence units appear in the selected context. These traces need not become a single combined score. Keeping them separate often makes the next action clearer, such as retrieving an exception paragraph or inspecting a missing definition.

Case C: a cutoff can turn an observed later hit into a scored zero

The third fictional ranking has its first relevant item at position four. Under cutoff three, no relevant item appears in the selected prefix, and the declared reciprocal-rank-at-three score is zero. Under cutoff five, the first relevant item is included and the score is one fourth. The full authored list did not change; only the scoring boundary changed.

A zero at cutoff three does not mean the document collection has no relevant material. This case explicitly contains a relevant fourth item. It also does not mean the retrieval engine returned no results, because the selected prefix contains three items. State the precise result: the first three positions contain no relevant item under the supplied judgments and the declared cutoff rule.

The downstream context boundary may differ from the evaluation cutoff. If a report evaluates five positions but the application passes only three, the one-fourth score describes a broader list than the assistant receives. Review the actual selected identifiers rather than assuming the metric's boundary and the prompt builder's boundary match. A change in text budget can alter that relationship without changing the stored ranking.

When comparing reports with different cutoffs, label both values explicitly and preserve the longer ranking if available. A first relevant item outside a shorter cutoff is useful diagnostic evidence, but it must not be counted inside that cutoff. Conversely, a missing tail in an exported list cannot be assumed irrelevant merely because the scoring script did not receive it.

Case D: query weighting changes the mean

The fourth group assigns Cedar a first relevant rank of one and Maple a first relevant rank of four. Their reciprocal ranks are one and one fourth. With one equal contribution per query, the mean is five eighths. Now suppose an entirely fictional workload gives Cedar one occurrence and Maple three occurrences. The occurrence-weighted mean becomes seven sixteenths.

The weighted calculation is one times one plus three times one fourth, divided by four occurrences. The underlying per-query ranks stay the same. The smaller result comes from giving more weight to the lower reciprocal value. This is a change in aggregation inputs, not evidence that the ranking method worsened or that a particular user had a worse experience.

Actual workload weighting requires actual workload evidence and a defined period. The one-to-three weights in this worksheet are invented teaching inputs. They do not describe traffic, user priorities or a recommended deployment mix. If chosen importance weights are used instead, call them chosen importance weights rather than observed frequency. That wording keeps the assumption visible in the resulting summary.

Show the unweighted query table even when a weighted aggregate is needed. It helps a reviewer see whether a headline change follows from ranking movement, query composition or weighting. If a new workload adds many repetitions of one query type, a changed mean might occur with every individual ranking unchanged. Without row-level evidence, that possibility is easy to miss.

Case E: ties require an order rule before a first rank exists

The fifth group supplies two candidates with equal authored scores. Candidate A is irrelevant and candidate B is relevant. One explicitly supplied order places A before B, giving first relevant rank two and reciprocal rank one half. Another explicitly supplied order places B before A, giving rank one and reciprocal rank one. The score values alone do not identify which order the actual system would use.

This exercise compares two declared orders rather than implementing a tie-breaking evaluator. It does not claim that a library is nondeterministic, that a production system uses alphabetical identifiers or that one tie policy is universally preferred. The practical requirement is to document how the ranked output was produced when equal scores occur, and to preserve the actual output order used for evaluation.

A stable sort can preserve input order among ties, but input order itself may change when a collection snapshot or upstream query changes. If evaluation depends on that behavior, save the relevant ordering fields. Re-running a score calculation from an unordered set can create a different first-hit rank even though the candidate scores and relevance labels look identical in a spreadsheet.

For an AI handoff, avoid claiming that the relevant item was always first merely because one tie arrangement shows it there. The application may select a bounded prefix in a specific order, and that order determines whether the item is included. Review the concrete list passed onward. A tied-score table is a useful diagnostic starting point, not a substitute for the ranked handoff evidence.

Case F: unknown first hits are not completed zero scores

The sixth group has three fictional queries. Cedar's first relevant rank is one, Maple's is two and Birch's result is unresolved. The mean over the two completed query rows is three fourths. Completed-row availability is two thirds. These fractions describe different things: reciprocal position among reviewed rows and the share of rows whose first-hit result has been established.

If Birch is later established to have its first relevant item at rank four, the full equal-query mean becomes seven twelfths. If Birch is instead established to have no relevant item within the declared cutoff, the full mean under the zero rule becomes one half. Those are hypothetical alternatives until the unresolved row is reviewed. Neither is the final observed mean for the three-query set.

An unresolved row may reflect missing ranking data, unfinished relevance review or an unspecified cutoff. The worksheet does not establish which cause applies. Keep a reason field when actual review evidence supplies one, and do not use zero as a convenient placeholder that participates in arithmetic. A zero score asserts a completed bounded evaluation under a convention; an unknown value says the necessary result is unavailable.

A report should state both the included query count and the known total planned query count. If only completed rows enter a mean, label that selection clearly. Otherwise a reader may compare a partial mean with a previous full-set mean and attribute the difference to a ranking change. The missing-row state can be more important than the apparent movement in the displayed aggregate.

Inspect first-hit changes at the query level

For a proposed system comparison, align the same query identifiers and review old and new first relevant ranks side by side. Record the collection snapshot and judgment version for each side. If one side uses a revised document collection, the comparison may be useful, but it no longer isolates a ranking-order change on a fixed set of candidates. State the actual scope rather than implying a controlled experiment.

Sort the comparison into improved, unchanged, worsened and unresolved first-hit states using the declared cutoff rule. Inspect a few concrete rows from each category, especially changes involving task-critical qualifications. Equal means can conceal offsetting improvements and regressions. Conversely, a higher mean can coexist with a regression on a query that matters to the intended workflow.

Do not convert a reciprocal-rank difference into time saved without separate evidence. The score scale encodes position through a reciprocal transformation. It does not include reading speed, document length, interface behavior or how much verification the reader performs. A first relevant result could be long and difficult to interpret, while a later short paragraph might answer a narrow question directly.

Connect a retrieval review to the answer stage through actual evidence, not a presumed relationship. Check whether the selected passage reaches the prompt with its heading, date, source identity and needed exception. Then review the assistant's claims against that supplied material if a real answer is available. This guide contains no such answer run, so it leaves that downstream quality question open.

Keep the result reproducible without overstating it

A compact result packet can include query identifiers, ranked item identifiers, relevance labels, first-hit positions, cutoff, tie policy and aggregation weights. Add the raw reciprocal fractions before rounding. That packet supports a precise explanation of the metric while leaving source validity and operational suitability to the reviews that actually address them.

If an AI assistant drafts the explanation, ask it to name the included rows and unresolved rows before summarizing the aggregate. Require it to distinguish a changed rank from a changed query mix. Ask for a sentence describing what the metric does not measure in this specific task, such as coverage of all required source paragraphs. The purpose is to preserve the scope of the supplied evidence through prose.

The downloadable cases below are original fictional teaching data. Their checks verify supplied positions, first-hit membership and exact arithmetic under declared conventions. They do not test a general retrieval evaluator, inspect a real collection, validate human judgments or measure user satisfaction. A useful conclusion names the specific next uncertainty, such as reviewing a tied output order or completing an unresolved query row.

Frequently asked questions

Does MRR measure the percentage of correct AI answers?

No. Under this guide's convention it averages reciprocal positions of the first relevant retrieved item. An AI answer requires separate evidence about claims, source use and task requirements. The first relevant result can be insufficient, ignored or misinterpreted even when its rank is favorable.

Why is one divided by average rank different?

The reciprocal transformation happens before the mean in MRR. Ranks one, two and four give seven twelfths when their reciprocal values are averaged. Their average rank is seven thirds, whose reciprocal is three sevenths. Those operations answer different numerical questions and cannot be substituted.

What does zero at a cutoff establish?

Under the declared teaching rule, it establishes that no relevant item appears inside that evaluated prefix. It does not establish that the collection has no relevant material, that the system returned no results or that an unresolved judgment is negative. Preserve the cutoff and review state beside the score.

Can two rankings have equal reciprocal rank but different coverage?

Yes. If their first relevant item is second, both score one half even when one ranking contains additional relevant items later. Inspect required evidence coverage separately when a task needs several passages. Equal first-hit values do not establish equal source content or equal downstream answer quality.

How should tied candidate scores be handled?

Record the order rule and the actual ranked output used for evaluation. The fictional A/B tie can place the first relevant item at rank one or two depending on the supplied order. This example does not prescribe a universal tie policy or reproduce an external evaluator's behavior.

Should an unresolved query be assigned zero?

Keep it unresolved until the evidence supports a bounded result. Zero is a completed scoring outcome under a convention, while an unknown row lacks the necessary information. A completed-row mean can be reported with its inclusion count and availability, but it should not masquerade as the full-set result.