Read Graded NDCG Through Gains, Discounts and the Ideal Candidate Order

A graded retrieval report can distinguish a highly useful source from a partly useful one instead of labeling both simply relevant. Normalized discounted cumulative gain, usually abbreviated NDCG, combines those grades with position discounts and an ideal-order comparison. Each part introduces a choice. To interpret the number, a reader needs the grade meaning, gain transformation, cutoff, ideal candidate set and treatment of equal prediction scores.

This guide develops a bounded reading method through original fictional rankings. It executes no retriever, model, scikit-learn function or client evaluation. The examples use declared nonnegative grades and local arithmetic to illustrate particular choices. They do not validate relevance judgments, measure user satisfaction or establish that an AI assistant produced a supported answer from the ranked material.

Separate the grade from the score used to order candidates

A retrieval system can give candidates numerical scores for ordering. An evaluator can separately assign relevance grades that express a relationship to a question. Those are different fields. The highest prediction score does not become the highest reference grade merely because the system placed the item first. Without that distinction, an evaluation can accidentally use the ranking's own preference as the evidence that the ranking is good.

Define grades in words before assigning numbers. A fictional scale might distinguish no support, partial support and direct support for a named question. That still requires source review and a consistent rubric. The arithmetic can accept a grade without establishing that a reviewer applied the rubric accurately or that another reviewer would agree.

Keep the query, source identifier, version, predicted score and reference grade on the same row. A source can receive different grades for different questions. A document that supplies background for one question might directly answer another. Do not carry a grade across queries as though it were a permanent quality rating of the source or its author.

The official scikit-learn ndcg_score interface describes reference scores ranked by predicted scores, logarithmic discounting, ideal normalization and cutoff options. It also warns that negative reference values can fall outside the familiar zero-to-one interpretation. That primary context is useful for reading a report, but this guide does not reproduce the library's full behavior or claim that every NDCG implementation uses identical gain and tie rules.

Name the gain and the discount explicitly

In the first teaching convention here, gain equals the supplied grade. A candidate at one-based position r contributes its gain divided by log base two of r plus one. Position one therefore has denominator one, position two has denominator log base two of three, and position three has denominator two. Sum the contributions through the declared cutoff to obtain discounted cumulative gain, or DCG.

The logarithmic formula is a declared scoring convention. It is not an observed measurement of how quickly a person loses attention as they read. A lower contribution at a later rank describes the mathematical discount, not a tested model of reading effort. Keep claims about actual user behavior separate unless a study or direct measurement supports them.

Some reports transform grades before applying discounts, for example using two to the grade power minus one as gain. That transformation changes how grade differences influence the total. A grade-three item then has gain seven rather than three. The choice should be stated; it is not a formatting detail that can be ignored when comparing two scores.

Preserve the unrounded contributions when checking a small example. A displayed decimal can help communication, but the formula and inputs are the reproducible record. For these teaching cases, local calculations use a stated numerical tolerance for logarithms and exact fractions where no logarithm is needed. They are not general implementation conformance tests.

Normalize against an ideal order from the same declared candidates

Ideal DCG applies the same gain rule, discount and cutoff after arranging the declared reference grades from highest to lowest. NDCG divides the actual DCG by that ideal value when the denominator is positive. The comparison therefore depends on what candidates belong to the ideal inventory. Removing an important missing candidate from that inventory can change the denominator as well as the apparent interpretation.

An ideal order is ideal for the supplied grades and convention, not a proof of a perfect answer. The source labeled highest may still omit a necessary qualification, and several sources may need to be combined carefully. A normalized value of one means the ranking reaches the convention's ideal gain arrangement for those inputs; it does not certify truth, permission, completeness or deployment readiness.

Apply the same cutoff to both actual and ideal calculations. If only one result position is being evaluated, the ideal denominator must describe that same one-position opportunity under the chosen candidate set. Using an ideal total from a longer list would answer a different question. Record k beside both totals so that the comparison can be checked.

A report should also say how it treats a query with no positive ideal gain. Raw zero divided by zero is undefined. Software can adopt a documented summary convention, but the convention is a choice that needs a label. This guide leaves the raw ratio undefined in its all-zero teaching case rather than inventing a universal default.

Case A: calculate a small graded ranking

The first fictional group gives candidates A, B and C grades three, one and zero. The actual order is B, A, C. Under linear gains, the actual DCG is one plus three divided by log base two of three. The ideal order is A, B, C, whose ideal DCG is three plus one divided by log base two of three.

Dividing those two positive totals gives the normalized value. The worksheet retains the formulas and checks their local numerical calculation. The important interpretation is that the actual order places the grade-one candidate before the grade-three candidate. The score describes that ordering relative to the authored ideal; it does not explain why a retriever would produce the order.

Both orders contain the same three candidates and the same grades. A difference in the total arises from their placement under unequal position discounts. This isolates one arithmetic relationship for teaching purposes. It is not a controlled experiment on a real system, and there is no actual source content or observed answer in the exercise.

When reviewing a real row-level result, ask which candidate changed position and why its grade matters to the question. A normalized decimal alone hides that explanation. An item-level trace can reveal that an important exception moved later, but establishing the exception's relevance still requires inspection of the actual source and evaluation rubric.

Case B: gain transformations can change the result

The second group uses the same grades three, one and zero, but orders them as zero, three and one. Linear gain produces actual DCG of three divided by log base two of three plus one half. The ideal linear total remains three plus one divided by log base two of three. The resulting ratio follows directly from those declared terms.

Now use the explicitly different gain transformation two to the grade power minus one. The grades become gains seven, one and zero. The actual transformed-gain total is seven divided by log base two of three plus one half, while the ideal total is seven plus one divided by log base two of three. The normalized result differs even though the candidate order and reference grades did not change.

Do not interpret that difference as a model improvement, an annotation correction or evidence that one gain rule is universally superior. It reflects a different weighting of grade distinctions. A project should explain why its rubric and gain rule fit the intended evaluation question, rather than choosing whichever formula yields a more attractive aggregate.

A comparison sheet can therefore have separate columns for raw grade and transformed gain. That makes it possible to inspect the choice without overwriting the original judgment. If a report gives only gains, a reader may not be able to recover the original rubric. Preserve both when the transformation matters to interpretation.

Case C: cutoff changes both the actual and ideal opportunity

The third case returns to the first group's actual order B, A, C, with gains one, three and zero. At cutoff one, actual DCG is one and ideal DCG is three, so the normalized value is one third. At cutoff two, the two positive-gain contributions enter both totals. The resulting ratio matches the full three-position calculation because the third candidate has zero gain.

This is a property of the authored list. It does not imply that cutoff two always captures a complete evidence set or that later candidates are generally unimportant. A different grade inventory could produce a different relationship. Keep the list and cutoff attached to any observation about where the score stops changing.

The context passed to an AI assistant may use a different boundary from the evaluator's k. A report evaluating two positions does not establish which paragraphs reach a prompt after filtering, token budgeting or deduplication. Review the actual handoff identifiers and source context separately. A metric boundary should not silently become a claim about application behavior.

If a project compares values at several cutoffs, show each actual and ideal total with its own k. That table makes the changing denominator visible and prevents a misleading comparison between one actual prefix and another ideal length. It also helps a reviewer distinguish a ranking change from a reporting change that only selects a different cutoff.

Case D: rescaling and shifting grades are different operations

The fourth group uses the first case's linear grades and actual order, then multiplies every gain by two. Actual DCG and ideal DCG both double. Their ratio remains unchanged while the raw totals change. This narrow invariance follows from using the same positive multiplicative scale in numerator and denominator under the declared linear-gain convention.

Adding one to every grade is different. The formerly zero-gain candidate now contributes a positive amount, and every included position adds another discounted unit. The normalized ratio can change. A zero-based grade scale and a shifted scale should therefore not be treated as interchangeable labels merely because their order is the same.

Neither operation establishes that the new numbers better represent relevance. Grade meaning belongs to the rubric and source review. A score that is numerically invariant under one transformation does not validate that transformation as a semantic decision. Preserve the original grades and explain any rescaling before comparing the results.

This distinction matters when importing a survey-like rating into a ranking metric. A blank, a minimum rating and a zero gain may represent different things. Do not shift or fill them simply to make a calculation run. Ask what the labels mean, which values are permitted and what the evaluator's gain rule expects.

Case E: tied predicted scores need an explicit treatment

The fifth fictional group has two candidates with equal prediction scores and reference gains three and one. At cutoff one, a supplied order that puts the grade-three item first reaches normalized gain one. An alternative supplied order that puts the grade-one item first reaches one third. The identical prediction scores do not choose between those orders by themselves.

If this exercise averages the two ordering outcomes equally, the average is two thirds. That is an explicitly authored two-order calculation. It is not a call to scikit-learn and does not claim to reproduce every aspect of a library's treatment of ties. Other tie groups and cutoffs would require their own stated treatment rather than extrapolating a complete implementation from this example.

The official interface includes an ignore_ties setting that assumes the absence of ties for efficiency. A reader should check the actual score data and evaluation setting before relying on that assumption. Equal numbers can arise through a model, a quantized export or display rounding, but this guide does not establish which cause applies to any actual report.

Keep raw scores available at sufficient precision when ties matter. Two displayed values that look equal may differ internally, and an exported rounded table may create ties not present in the original evaluation. Conversely, an arbitrary stable row order can hide a real tie group. Preserve the actual evaluation inputs before attributing a difference to the ranking method.

Case F: zero ideal gain and unknown grades are separate states

The final group first supplies an all-zero gain inventory. Actual DCG is zero and ideal DCG is zero under the declared convention. The raw normalized ratio is therefore undefined. This guide does not replace it with a value and call that measured relevance. If a software report uses a special value, identify its documented convention and query inclusion rule.

The same group separately includes a candidate whose grade is unknown. That is not an all-zero inventory. An unknown grade leaves both its contribution and potentially the ideal order unresolved. Filling it with zero would assert a relevance judgment that has not been made and could alter the numerator, denominator and interpretation together.

A useful report can count completed judgments while leaving the score open. Two known grades out of three is a judgment-availability fraction, not NDCG, source coverage or confidence. Keep the status visible beside any partial calculation, and do not compare a completed-subset score with a full-set score without naming the changed population.

If an AI assistant drafts an explanation, require it to separate the zero-denominator case from the missing-judgment case. It can identify the next information needed, such as a grade or a declared software convention. It must not invent a label merely to produce a normalized decimal for every row in a report.

Read the aggregate after inspecting individual queries

A multi-query report introduces another aggregation step. Equal-query averages, observed-frequency weights and chosen-importance weights answer different questions. Keep the per-query NDCG values, included query count and any weights with the summary. A higher mean can arise from a changed query mix even when every individual ranking remains unchanged.

Look at regressions and unresolved queries as well as improvements. A favorable mean can conceal a source moving down for a query important to the workflow. The metric alone cannot decide that priority. Review the actual task, source role and consequences of missing evidence rather than converting a small aggregate gain into an automatic release decision.

When a change affects collection membership, reference grades and ranking order together, list those differences explicitly. The new result may still be useful, but it does not isolate one cause. A clear comparison states which conditions stayed fixed and which changed, leaving causal or operational conclusions to evidence that actually supports them.

The worksheet below is a small arithmetic aid to interpretation. It checks declared gain formulas, logarithmic sums, candidate order, cutoff boundaries, scale relationships and missing states. It does not validate a rubric, inspect source contents, run a ranking system or measure an AI answer. A useful review finishes with a precise explanation of the unresolved source or workflow question.

Write a result sentence that preserves the definitions

A reproducible result sentence can state that a particular ordered list reached a specified ratio under linear gains, a named logarithmic discount and a fixed cutoff. Include the candidate inventory and judgment version beside it. That sentence is narrower than saying the assistant found the best answer, but it gives a reviewer enough information to check the actual calculation.

When two reports disagree, compare their gain rules, ideal inventories, cutoffs, weights and tie handling before looking for a model explanation. One report might normalize against a filtered candidate set while the other includes additional high-grade items. Another might use a transformed gain scale. Those differences can be identified from the evaluation record without asserting that either source judgment or implementation is wrong.

Keep the explanation tied to the observed fields. A changed denominator is evidence of a changed calculation condition, not evidence that the application became less useful. An unchanged normalized ratio can coexist with different raw totals, and a changed ratio can follow from a revised grade rubric. The next review should address the specific condition that changed rather than treating every numerical movement as an operational outcome.

Frequently asked questions

Is a reference grade the same as a predicted ranking score?

No. The score orders candidates, while the reference grade expresses the supplied relevance judgment for the question. Keep both fields and their roles distinct. Using the ranking's own preference as its reference would not independently evaluate whether the source addresses the task.

Does NDCG of one mean the AI answer is correct?

No. It means the ranking reaches the declared ideal gain arrangement under the supplied grades, candidate set and convention. It does not establish source truth, permission, complete evidence or correct use by an assistant. Answer quality needs a separate review of actual claims and sources.

Why must the gain transformation be stated?

Linear grade gains and transformed gains weight grade differences differently. The second example changes the normalized result without changing order or raw grades. Preserve the rubric and transformation instead of treating every report labeled NDCG as directly comparable.

Should the ideal total use the same cutoff?

Yes, for the convention defined here. Actual and ideal totals describe the same number of evaluated positions. A different ideal length changes the question. Keep both totals with k so a reader can reproduce the normalization and distinguish a cutoff change from a ranking change.

What does an all-zero ideal inventory mean?

Its raw ratio is zero divided by zero and is undefined under this guide's arithmetic. A tool may use a documented convention, which should be named. Unknown grades are a different state and must not be silently replaced with zero to make the calculation produce a value.

Can the worksheet certify my evaluator?

No. It checks a few authored inputs and declared formulas using local arithmetic. It does not test general conformance, execute scikit-learn or validate a real relevance rubric. Use it to identify the definitions and evidence needed for a more specific review.