Read Retrieval Precision and Recall at a Cutoff Before Comparing AI Context

A retrieval report can show a high percentage while leaving the most important source outside the material passed to an AI assistant. Precision at a cutoff and recall at a cutoff answer different counting questions. Reading them requires knowing what one retrieved item represents, which items were judged relevant, how the ranking was shortened and whether the relevant set is actually known. The metric name alone does not supply those decisions.

This guide develops a practical way to read those counts before using a retrieval result in an AI workflow. It uses original fictional document identifiers and deliberately small lists. No search engine, embedding model, retriever, language model or client dataset was executed. The numerical examples are supplied teaching inputs, and their arithmetic does not establish that a real document supports an answer, that a relevance judgment is correct or that a system is suitable for deployment.

Start with the question and the retrieval unit

Write the question in the form a reader would actually ask. Then describe the unit being ranked. A whole policy document, a numbered paragraph, a page image and a text chunk can each occupy a result position, but their counts have different meanings. Three chunks from one document are three chunk occurrences. They are not automatically three independent sources or three different parts of the answer.

For a fictional question about an exception in Workshop Note E4, suppose the retrieval unit is a versioned paragraph. The expected evidence might include the exception paragraph and the paragraph that defines the affected request category. If the unit changes to entire files, a file containing both paragraphs could become one relevant item. A report should not compare those two counts as if the underlying task had stayed identical.

Keep retrieval relevance separate from permission to use a source. A paragraph might address the question and still belong to an outdated version or an inaccessible collection. Conversely, an authorized document may contain nothing useful for this particular question. Record the eligibility rule used before ranking, the relevance rule used for evaluation and the evidence needed for the downstream answer as separate decisions.

A practical review sheet can name the query, collection snapshot, item identifier, version, rank and binary relevance judgment. Add a place for unresolved judgments. That last field matters because an empty review cell does not mean the reviewer found an item irrelevant. Without it, a tidy binary table can quietly convert unfinished work into negative evidence and change both numerators and denominators.

Define the cutoff before reading a percentage

In this guide, a result list is already in its declared order and contains distinct item identifiers unless a case explicitly says otherwise. The cutoff k is a positive integer. When the list supplies at least k items, precision at k is the number of relevant items among its first k positions divided by k. Recall at k is that same relevant count divided by the number of relevant items in the declared evaluation collection.

The two measures share a numerator but generally use different denominators. Precision asks about the selected result positions. Recall asks about the relevant collection items the selection reaches. Neither ratio directly measures whether the assistant wrote a supported answer. An assistant could overlook a retrieved paragraph, combine incompatible versions or make a claim absent from every supplied source even when the retrieval counts look favorable.

The official TorchMetrics retrieval interfaces are useful primary documentation for seeing that query grouping, cutoff handling, aggregation and empty-target behavior are explicit options. Its precision interface also exposes adaptive cutoff behavior. Those interface details are not a universal evaluation policy. This guide does not execute TorchMetrics, reproduce its complete behavior or claim that an unspecified platform uses its defaults.

Before comparing reports, require a short sentence that states the cutoff and denominator convention. For example, the authored exercise might say that k equals three, the ranking contains five distinct paragraph identifiers, and the relevance inventory contains four identifiers. This makes the fraction interpretable without asking a reader to guess whether three means requested positions, returned positions or relevant sources.

Case A: one numerator, two different questions

The first fictional case has relevant identifiers A, C, E and G. Its ordered results are A, B, C, D and E, with k equal to three. The first three positions therefore contain A and C as relevant items and B as an irrelevant item. The relevant count at the cutoff is two. Precision at three is two thirds, while recall at three is two fourths, or one half.

The difference is not a disagreement between metrics. Two of the three inspected result positions are relevant, and two of the four declared relevant identifiers have been retrieved within that cutoff. Both statements describe the same authored selection. Reporting only two thirds could hide the fact that half the relevant inventory remains outside the selected prefix; reporting only one half could hide that one selected position is irrelevant.

If an AI context builder passes only those first three positions, the fifth result E is not part of that supplied context. Its presence somewhere in the longer ranking does not establish that the assistant could use it. Name the actual handoff boundary. A search interface showing five links and a model receiving three chunks are different objects, even if both are casually called the retrieval result.

A useful reviewer response would ask which omitted relevant item is needed for the intended answer. The arithmetic treats A and C equally, but the task may require a qualification contained only in E. That question needs source inspection and a task definition. The two fractions cannot determine importance, replace evidence review or establish that the remaining items are unnecessary.

Case B: increasing the cutoff changes the tradeoff

The second case uses relevant identifiers A, C and E with the ordered results A, B, C, D and E. At cutoff two, only A is relevant, giving precision one half and recall one third. At cutoff five, all three relevant identifiers are included, giving precision three fifths and recall one. Here both ratios increase, but that pattern is a property of this list rather than a rule about larger cutoffs.

For contrast, look within the same supplied ranking at cutoff one. Its precision is one and its recall is one third. Moving to cutoff two adds the irrelevant B, so precision falls to one half while recall stays one third. Moving farther through a distinct fixed ranking can add relevant coverage without guaranteeing that the proportion of relevant selected positions rises.

Choose a cutoff around the actual workflow question. A reader who can inspect two documents has a different selection boundary from a context builder allowed five paragraph slots. The metric should describe that boundary rather than select it automatically. More retrieved text could increase source coverage while increasing the work required to inspect relevance, reconcile versions and preserve qualifications in the final prompt.

When comparing cutoffs, show the count sequence alongside the ratios. The authored list has relevant-prefix counts one, one, two, two and three across positions one through five. That small sequence explains each movement. It also exposes a common reporting problem: a headline can improve because the cutoff changed, even though no ranking order, source content or relevance judgment changed at all.

Case C: short result lists need a stated rule

The third case requests k equal to five but supplies only A and B. Only A is judged relevant. Under a declared fixed requested-slot denominator, the illustrative precision is one fifth. Under a declared returned-item denominator, it is one half. These are different accounting conventions for an undersized result list. Neither should appear under an unexplained label that invites the reader to assume the other.

The official precision interface documents an adaptive cutoff option, which is a concrete reminder to inspect implementation settings. This exercise calculates the two authored conventions directly. It does not diagnose a software defect or claim that every evaluator should choose the same treatment. The right report identifies its rule, how many results were requested and how many eligible results were actually available.

A short list can arise for many reasons, including a small eligible collection, filtering or an incomplete retrieval process. The two identifiers supplied here do not establish any of those causes. Do not write that the system failed to retrieve three documents unless the intended eligible population and actual process evidence support that conclusion. A missing position is an observation about this supplied list, not a complete operational explanation.

In an AI workflow, carry the short-list state into the handoff. If the prompt template assumes five sources, two returned items should not silently become five through duplication or empty placeholders. The downstream reader needs to know that the available evidence list is shorter than requested. That factual statement can remain useful even when the final precision convention is still under discussion.

Case D: repeated chunks do not create new source coverage

The fourth case declares relevant identifiers A and C, then supplies the ordered occurrences A, A and B at cutoff three. If occurrences are counted independently, two of the three positions carry a relevant label. However, only one distinct relevant identifier appears. Dividing two occurrences by two distinct relevant identifiers would misleadingly suggest complete distinct-item recall. Those numerator and denominator units do not match.

For this teaching case, distinct relevant coverage is one of the two relevant identifiers, or one half. The occurrence-level relevant proportion is two thirds. Both can be recorded if their units are explicit. A report that needs distinct-item precision must first define how repeated occurrences affect rank and cutoff, rather than deduplicating after the calculation and retaining the original denominator without explanation.

Real chunking can make the relationship more complicated than this exact duplicate. Overlapping chunks may share most words while retaining different qualifiers; separate document copies may have distinct version authority. A stable identifier is evidence about the defined unit, not proof that two passages are semantically identical. Inspect the relationship before deciding that an overlap is redundant or that one chunk can safely replace another.

A practical context review should therefore count source units and occurrences separately. Keep the original ranked occurrence list available, record any deduplication rule and show the resulting list if it changes the handoff. This preserves the reason that a candidate moved into or out of the first k positions. It also helps prevent a repeated paragraph from occupying the entire apparent evidence budget.

Case E: averaging queries changes the denominator story

The fifth case has two fictional queries. Cedar has one relevant item in a selection of two and one relevant item in its declared collection. Maple has one relevant item in a selection of two and nine relevant items in its declared collection. Each query therefore has precision one half. Cedar recall is one, and Maple recall is one ninth.

The equal-query mean recall is five ninths: add one and one ninth, then divide by two. Pooling the retrieved relevant counts and declared relevant counts instead gives two tenths, or one fifth. The difference is caused by weighting. The equal-query mean gives Cedar and Maple equal influence; the pooled ratio gives their relevant-item inventories influence through the denominator.

Neither number tells a reader which query deserves more business attention. Those priorities require a stated use case, and any weights should be reported as choices rather than discovered truths. A production workload might repeat some query types frequently, but this exercise contains no observed traffic. Calling an equal-query average representative of real users would add evidence that the authored table does not contain.

Show the per-query rows before the summary when the inventories differ substantially. A single mean can hide Maple's low relevant-item coverage even though it is mathematically correct. The response should be to understand the task and source inventory, not to replace the mean with whichever aggregation produces the more attractive headline. Preserve both definitions when explaining why reports differ.

Case F: unknown relevance keeps the result open

The sixth case supplies three ranked items with judgments relevant, irrelevant and unknown. At cutoff three, the known relevant count is one and the unresolved count is one. If the unknown item is later judged relevant, precision under the three-position convention becomes two thirds. If it is judged irrelevant, precision becomes one third. Until that judgment exists, neither alternative is the measured final precision.

The fraction of completed judgments is two thirds, but that is a review-availability count. It is not precision, recall, semantic coverage or a confidence level. Reporting it beside the open result helps a reader distinguish the amount of finished labeling from the retrieval behavior being evaluated. The unknown field should remain visibly different from a negative judgment in exported tables.

Recall has an additional prerequisite: a defensible inventory of relevant items for the declared collection. Knowing the labels of three returned items does not establish how many relevant items remain elsewhere. If the total relevant set is unknown, avoid presenting a recall fraction with an invented denominator. A locally useful inspection count can still be reported without renaming it collection-wide recall.

An AI assistant can help assemble a review table from supplied identifiers, but it should preserve unresolved cells and cite the fields it used. Ask for the relevant count, cutoff convention, distinct-item rule and missing judgments before requesting prose about retrieval quality. A polished paragraph must not turn an incomplete reference inventory into a claim that all needed evidence was found.

Connect retrieval counts to an answer review

Once the counts are clear, inspect how the selected material enters the actual task. Does the supplied text include headings, dates and exceptions? Are page images needed to interpret a figure? Did a collection filter remove a version that the question explicitly requested? These are source and workflow questions. Precision and recall do not identify which context relationship was lost.

Keep a small trace from query to selected item to proposed answer claim. The trace can point to the exact paragraph used and record a qualification that must survive summarization. If a claim lacks a supporting selected passage, leave it unresolved even when the retrieval report looks strong. Conversely, a poor aggregate metric does not prove that every individual answer is unsupported; inspect the specific evidence chain.

Compare changes on the same queries and declared inventory whenever that comparison is the intended question. Retain old and new ranked lists rather than only their aggregate ratios. A changed result could reflect a new source snapshot, revised relevance labels, a different cutoff or a genuinely different order. Separate those changes before attributing an apparent improvement to the retrieval method.

The original worksheet below is intended for that kind of bounded inspection. Its checks establish literal membership, occurrence counts and exact fractions for the supplied cases. They do not validate the relevance labels, inspect any actual document collection or predict how a language model will use retrieved text. A useful next action follows from the unresolved field, such as reviewing an item judgment or defining the handoff cutoff.

Write a comparison that another reviewer can reproduce

A concrete comparison note should name the evaluation question before stating a change. For the authored examples, a reproducible note might identify the same collection, the same binary judgments and the same ordered item list, then explain that only the cutoff changed. The resulting fraction difference describes selection size under those inputs. It does not establish a better retriever, because no retriever changed in that comparison.

If judgments change instead, preserve both label versions and the reason for revision. A source previously marked relevant may turn out to address a different version of the question. Recomputing a metric can be appropriate, but the new score should not erase the old evaluation conditions. Keep the interpretation close to the changed evidence: a revised judgment table produced revised arithmetic, with operational implications still requiring review.

When several conditions change together, write separate rows for collection snapshot, query text, item unit, filtering rule, ranking order, cutoff and judgment version. This is a compact way to identify what a proposed explanation would need to establish. It also gives the next reviewer a practical starting point for checking the comparison without reconstructing the whole project from a headline number.

Frequently asked questions

Is precision at k the same as answer accuracy?

No. In this guide it counts relevant retrieved items within a declared prefix. Answer accuracy requires a separate definition and review of what the assistant actually claims. A relevant passage can be ignored or misinterpreted, and a high relevant-item proportion does not certify an answer.

Can recall at k be calculated from returned items alone?

Not without an appropriate relevant-item denominator. Returned-item judgments identify relevant members of that selection, but they do not necessarily identify all relevant collection items. Keep recall unresolved when that reference inventory is unknown, and name any narrower inspection count accurately.

Should a short result list use k or the returned count?

State the convention explicitly. The fictional two-item list gives one fifth under five requested slots and one half under two returned items. Software settings and evaluation policy can differ. Preserve requested size, returned size and chosen denominator so another reader can reproduce the calculation.

Does a repeated relevant chunk increase recall?

It can increase a relevant occurrence count without reaching a new distinct source unit. The worksheet's repeated A reaches only one of two relevant identifiers. Define the retrieval unit and duplication treatment before computing coverage, especially when overlapping chunks or copied document versions are involved.

Why can average recall differ from pooled recall?

The weights differ. An equal-query mean gives each query equal influence, while pooled relevant-item counts give larger relevant inventories more denominator weight. The Cedar and Maple example produces five ninths versus one fifth. Show the per-query counts and chosen aggregation before interpreting either summary.

What should an AI assistant do with an unknown relevance label?

Retain it as unknown and explain which result remains unresolved. It can calculate explicitly hypothetical alternatives from supplied assumptions, but those alternatives must not become observed results. Completed-label availability is useful context, not a substitute for the missing judgment or a measure of answer quality.