Read Average Precision Through Relevance Positions, Thresholds and Recall Increments

Average precision describes how relevant items are distributed through a scored or ranked list under a particular evaluation convention. It is easy to confuse it with precision at one cutoff, a first-hit score or the area produced by connecting points on a graph. Those comparisons can all be mathematically meaningful, but they are different calculations. Before interpreting an AI retrieval report, identify the relevance inventory, score ties and aggregation rule that produced its average-precision number.

This guide uses original fictional rankings and manually supplied binary relevance judgments. No retriever, embedding model, language model, client system or scikit-learn function was run. Exact fractions check the authored arithmetic only. They do not validate the relevance labels, show how a real assistant used sources or establish that a particular ranking is suitable for an operating workflow.

Identify the query, collection and relevant unit

Start with one question and one declared evaluation collection. Decide whether an item means a document, a versioned paragraph or another unit. The relevant inventory must use that same unit. Three overlapping chunks from one document are not automatically three independent supporting sources, and a repeated occurrence should not silently become another distinct relevant item in the denominator.

A relevance judgment also needs a question. A paragraph can address the broad topic while failing to supply a required exception or version. Keep source eligibility, relevance to the question and support for the eventual answer separate. An item excluded by a permission rule and an eligible item judged irrelevant are different states, even if neither appears in the selected context.

Write the identifiers and judgments before calculating the summary. Keep unresolved judgments visible rather than substituting zero. If an evaluation collection contains only a reviewed subset of possible sources, describe that boundary. A score against a declared subset can be useful without establishing coverage of every relevant source in a larger organization.

A small result packet should identify the query, candidate collection, relevance version, score direction and retrieval unit. Those fields make a later comparison interpretable. Without them, a change in the average could reflect a different collection or labeling policy rather than a different ranking method. The metric name cannot supply the missing evaluation conditions.

Use a declared calculation rather than an ambiguous abbreviation

For a complete, distinctly ranked binary list with known relevant inventory, this guide's first exercise takes precision at each relevant position, sums those values and divides by the number of relevant items. A relevant item early in the list contributes a precision value calculated from a shorter prefix. Later irrelevant items do not add their own terms, but they affect the denominators of subsequent relevant positions.

The official scikit-learn average_precision_score documentation describes a non-interpolated threshold calculation that weights precision by increases in recall. It explicitly distinguishes that calculation from trapezoidal area under a precision-recall curve. That primary interface context is useful when reading software reports. This guide does not execute the function or claim that every tool using the abbreviation AP follows the same convention.

With distinct scores and a complete binary collection, the position-based teaching calculation can express the recall-increment idea using equal increments for relevant items. Tied scores require additional care because a threshold can include several candidates together. A row order chosen for display is not necessarily the threshold behavior used by a scoring implementation.

Before comparing numbers, ask for a definition in words and the actual denominator. Is the report evaluating a complete candidate collection, a truncated list with a separately known relevant total or a tool-specific task? Keep the answer with the result. Two correct calculations can differ because their conventions differ, without either one demonstrating a software defect.

Case A: read the precision at relevant positions

The first fictional ranking contains five distinct candidates whose supplied relevance labels, in order, are one, zero, one, zero and one. All three relevant items in the declared collection appear in the list. Relevant positions are one, three and five. The precision values at those positions are one, two thirds and three fifths respectively.

Adding the three terms gives thirty-four fifteenths, then dividing by the three relevant items gives average precision thirty-four forty-fifths. Precision at the final cutoff of five is only three fifths. The first-hit reciprocal rank is one because the first candidate is relevant. All three numbers describe the same authored ranking through different questions.

Do not call thirty-four forty-fifths the proportion of correct AI answers. No answer exists in the exercise. The number summarizes the supplied item ordering and labels. It says nothing about whether a retrieved paragraph contains the qualification needed for a particular answer, whether the assistant reads it or whether an output preserves its meaning.

A useful diagnostic table includes rank, item identifier, relevance, cumulative relevant count and precision for each prefix. Mark which rows contribute terms. A reviewer can then see why irrelevant positions affect later terms without being averaged as independent zero rows. That table is often more informative than a rounded headline alone.

Understand why irrelevant positions still matter

An irrelevant item before a later relevant item increases the size of that later prefix. With the same relevant count, the resulting precision is lower. Moving an irrelevant item after the last relevant item in a complete list need not change this particular average-precision calculation. This is a consequence of the defined terms, not a finding about the reading effort of a real person.

The position of each relevant item matters, but that does not mean all supporting evidence has equal practical value. A required exception may be more important to the task than a general overview. Binary labels suppress such distinctions. If the task needs graded relevance or coverage of specific evidence roles, inspect those relationships separately rather than expecting a binary summary to recover them.

Likewise, a source can be relevant while being long, difficult to interpret or superseded for operational use. Average precision does not measure reading time, source authority or permission. Keep those fields available for review. A ranking can improve under the supplied labels while leaving an important workflow constraint unchanged or unresolved.

When a report moves, examine which relevant items changed position. A single early movement and several later movements can produce a similar aggregate difference while creating different source-selection questions. Preserve item identifiers so that the discussion can refer to actual relationships, rather than interpreting a decimal increase as a complete explanation of the change.

Case B: the same first hit can conceal different later order

The second group compares two authored four-item lists. S has relevance labels one, zero, zero and one. T has labels one, one, zero and zero. Each contains two relevant items, starts with a relevant item and has final precision one half. Their first-hit reciprocal ranks are therefore equal at one, but their average precisions differ.

S contributes one at rank one and one half at rank four, producing three fourths after dividing by two relevant items. T contributes one at rank one and one at rank two, producing one. The later placement of the second relevant item explains the difference. Neither result depends on a new source judgment or a different relevant total in this exercise.

This is why a first-hit measure and average precision can complement each other. One emphasizes reaching the earliest relevant item; the other includes the positions of additional relevant items under its convention. They should not be merged into a single word such as accuracy. Keep the separate definitions visible when a task needs more than one source passage.

The exercise does not establish which list would yield the better AI answer. The actual source contents and required evidence roles are not supplied. The bounded conclusion is that the first-hit and final-cutoff measures are equal while the defined average-precision values differ. Any claim about answer quality would need another evaluation.

Case C: tied scores change the threshold question

The third fictional group supplies scores 0.9, 0.5 and 0.5 with binary labels one, one and zero. At the higher threshold, the first item alone is selected, giving precision one and recall one half. Lowering the threshold to include the tied pair selects all three items, giving precision two thirds and recall one.

Under the declared non-interpolated threshold convention, the two recall increments are each one half. The weighted precision sum is one half times one plus one half times two thirds, or five sixths. This is an authored calculation from two threshold groups, not an observed result from scikit-learn or a general evaluator conformance test.

If a display breaks the tie into an arbitrary sequential order, the position-based calculation can change. Putting the relevant tied item first gives relevant ranks one and two and average one. Putting the irrelevant tied item first gives relevant ranks one and three and average five sixths. A simple average of those two ordering results is eleven twelfths, which is a third number rather than the declared grouped-threshold calculation.

Do not use that example to assert that a particular platform mishandles ties. The exercise compares explicit conventions and shows why the report needs to name one. Save the candidate scores, actual order and tie treatment. A spreadsheet that sorts tied candidates by a convenient identifier may not reproduce the scoring process a software tool used.

Keep interpolation and display choices separate

A precision-recall plot can be drawn from threshold points, but the visual connection between them is an additional choice. A straight line and a step-based representation can suggest different areas. The official documentation's distinction between non-interpolated average precision and trapezoidal area matters because a shared graph title can hide that difference.

Ask whether a reported number comes from the metric function, a separate area calculation or a charting tool. Keep its label precise. A plot can be useful for inspecting the tradeoff across thresholds without every area inferred from the picture being the same AP value. This guide draws no curve and claims no complete implementation of either calculation.

Rounding is another display choice. Preserve exact fractions for these small teaching cases and state a precision when showing decimals. Two values that round alike can still differ, while two reports using different display precision can appear to conflict despite representing the same underlying result. Review the arithmetic before attributing a difference to ranking behavior.

For actual comparisons, retain the per-query calculation and its inputs. A screen image of a curve can help communication, but it usually cannot reproduce the original candidate membership and relevance judgments. Keep that evidence in a separate structured record so a later reviewer can understand what the displayed summary includes.

Case D: a truncated list needs its own denominator statement

The fourth case shows only the first three labels of a longer declared collection: one, zero and one. The relevant inventory for the full collection contains three items. Summing precision at the two relevant retrieved positions gives one plus two thirds. Under a declared truncated contribution divided by the full relevant total, the result is five ninths.

Dividing by only the two relevant retrieved items instead would give five sixths. That second denominator removes the unretrieved relevant item from the summary. The exercise demonstrates a difference in conventions, not a universal prescription for every metric labeled AP at a cutoff. Neither figure should appear without explaining which relevant total was used.

A short exported list may not contain enough information to calculate the intended complete-list AP. Do not assume the unseen tail is all irrelevant, and do not invent positions for missing relevant items. Preserve the cutoff, known full relevant count and available prefix. A narrower calculation can be reported accurately while the full result remains unavailable.

An AI context builder might select a prefix for reasons unrelated to the evaluator's original list length. Record that handoff boundary separately. A good complete-list average does not establish that the selected context contains every required passage, and a missing tail in a prompt is not repaired by mentioning that the retriever found it somewhere farther down.

Case E: query aggregation is a second calculation

The fifth group gives Cedar an authored per-query AP of one and Maple an authored per-query AP of one half. Their equal-query mean is three fourths. If a fictional weighting gives Cedar one occurrence and Maple three, the weighted mean becomes five eighths. The query-level values are unchanged; only their contributions to the summary differ.

The occurrence weights are invented and describe no real user traffic. If a project uses actual workload weights, preserve the observation period and query grouping rule. If it uses chosen importance weights, identify them as choices. A weighted average should not inherit a claim of representativeness merely because its arithmetic is exact.

Avoid pooling item rows from unrelated queries and sorting them together unless that is the explicitly intended metric. Candidate relevance is defined relative to its query, and scores from different queries may not have a shared comparison meaning. An item relevant to Cedar is not automatically relevant to Maple, even if both records contain the same document identifier.

Inspect per-query changes before interpreting a mean change. A higher average can coexist with a regression on one query, and a changed query mix can move the mean with all individual values unchanged. The summary supports a bounded statement about its included rows and weights. It does not identify which workflow deserves priority or establish a business outcome.

Case F: an unknown judgment keeps the full AP unresolved

The final fictional list has labels one, zero and unknown in that order. If the unknown third item is ultimately irrelevant, the list has one relevant item at rank one and the complete-list average precision is one. If it is ultimately relevant, the contributing precisions are one and two thirds, giving five sixths. Both alternatives are hypothetical until the judgment is established.

Dropping the unknown item produces a completed-subset calculation, but it changes the evaluated collection. Reporting that subset's value as the full-list result hides the unresolved evidence. Keep judgment availability separate from the metric. Here two of three labels are known; that two-thirds fraction is a review-status count, not retrieval precision or a confidence estimate.

An unknown relevance label is not the same as a known negative, and an absent candidate row is not the same as an unknown label on a known row. Preserve those states in exports. The next action might be to review a source or recover a missing list segment, but the supplied exercise does not establish which operational cause produced the gap.

When using an AI assistant to draft a report, ask it to state the unresolved row and the denominator convention before interpreting the number. It may calculate clearly labeled alternatives from supplied assumptions. It must not turn either hypothetical label into an observed judgment or turn a complete-subset result into evidence that the whole retrieval problem is solved.

Use average precision as one part of source review

A practical review connects a query, a ranked item and the answer claim that would use it. Check whether headings, dates and exceptions remain attached to the selected passage. A relevant-item ranking summary cannot identify every context relationship that might be lost between retrieval and generation. That requires inspecting the actual handoff and, when available, the actual answer.

For a proposed change, preserve old and new rankings, collection snapshots and label versions. If several conditions change together, list them separately before offering a cause. A new average can follow from revised labels, a different relevant inventory or a different tie convention. Those are meaningful changes, but they are not automatically improvements in the retrieval method itself.

The worksheet below supports six narrow comparisons in authored data. Its checks establish membership, precision terms, exact fractions, score-group boundaries and aggregation. They do not evaluate a production retriever, validate human relevance judgments or certify a downstream answer. A useful conclusion names what the arithmetic establishes and the next source or workflow question still needing evidence.

Keep one compact comparison record

Before circulating a result, name the query set, source snapshot, relevant inventory and calculation convention in the same note as the value. Include the row-level difference that explains any change. This gives another reviewer a concrete starting point and prevents an apparently precise headline from becoming detached from its evaluation conditions.

Frequently asked questions

Is average precision just precision at the final cutoff?

No. The first case has final precision three fifths but average precision thirty-four forty-fifths under its complete-list convention. The latter includes precision values at relevant positions. Keep the calculation and relevant inventory explicit rather than using the two names interchangeably.

Does AP equal trapezoidal precision-recall area?

Not under the official non-interpolated definition discussed here. The documentation distinguishes threshold-weighted precision from a linearly interpolated area. Identify which calculation produced the report, and do not infer an implementation from a graph's appearance or an ambiguous abbreviation.

Why do tied scores need special attention?

A threshold can include tied candidates together, while a displayed row order may separate them arbitrarily. The fictional tied group gives five sixths under the declared threshold convention, while ordered alternatives can differ. Preserve scores and tie rules before trying to reproduce a result.

Can a short retrieved list establish full average precision?

Not necessarily. A truncated prefix needs a stated relevant-item denominator, and the missing tail may contain relevant items. The worksheet contrasts five ninths using the full relevant inventory with five sixths using retrieved positives only. Those are declared alternatives, not interchangeable universal definitions.

Does a higher query mean prove a better AI answer?

No. The mean depends on included queries, judgments and weights, while answer quality requires separate evidence. Inspect query-level changes and the actual selected sources. The guide contains no model run, answer evaluation, workload measurement or business result.

What should happen to an unknown relevance label?

Keep it unknown and identify which full result remains unresolved. Hypothetical alternatives may be calculated explicitly, but they are not observed outcomes. Completed-label availability describes review status and cannot replace the missing judgment or validate the collection's relevance inventory.