Follow Paginated Results in an AI-Assisted Import

An AI-assisted import can produce a well-formatted table while retrieving only the first page of its source. The rows may be valid, the field names may be correct and every calculation on those rows may work. The unresolved question is whether the collection was traversed as intended. Pagination is the part of the interface that explains how to continue a list, and its rules need to be understood before a partial result becomes the basis for a summary or a record-writing workflow.

This guide uses six original fictional response chains to make that problem concrete. The local examples are static teaching data, not responses observed from a provider or AI model. Their token names, record identifiers and rules are invented. No real API is called, no credentials are used and no customer records are imported. The exercise helps a person describe a traversal, recognize its stopping condition and preserve an incomplete result without presenting it as a complete collection.

Read the particular interface before drafting the loop

Pagination is a family of interfaces rather than one universal field convention. An endpoint may return a continuation token, an explicit next link, a more-results flag or another documented mechanism. Its request may use a page number, an offset or a cursor. An assistant should work from the actual endpoint's documented contract instead of assembling a loop from familiar field names. Similar words across providers do not establish identical behavior or interchangeable requests.

For a bounded provider example, Stripe's list documentation describes starting_after and ending_before object-ID cursors and a has_more response flag. It says the two cursor parameters are mutually exclusive and notes that the v2 namespace uses a different pagination interface. This is context for the specific documented list interface, not the contract of the fictional examples below. Do not copy its cursor names into an unrelated API or assume a different provider shares its ordering and continuation rules.

Google's AIP-158 is API-design guidance with a different shape. It describes next_page_token, permits fewer results than requested even before the end, and treats an empty continuation token as the end signal under that guidance. It also discusses opaque tokens and keeping request arguments consistent, with a stated page-size exception. Those are specific guidance details, not guarantees about every Google endpoint or every paginated service. The practical lesson is to identify the applicable contract and its exceptions explicitly.

Write a short interface note before asking an assistant to draft implementation logic. Name the endpoint and version, initial request, result field, continuation field, terminal condition, permitted parameter changes and error behavior. If the documentation leaves a point unresolved, retain the question. A loop that compiles is not evidence that its assumptions match the service. The six teaching cases use their own declared rules so that their local results can be checked without suggesting compatibility with an actual provider.

Use a complete request scope, not just a token

A list request has a scope: parent collection, filters, ordering, fields, access context and any relevant time condition. A continuation token belongs to the traversal described by its contract. If the next request changes the filter, the resulting rows may no longer answer the same question. An assistant can preserve the token exactly while still changing the meaning of the request. Keep the scope beside the continuation state so that both can be inspected together.

For the fictional fixture, the declared scope is collection teaching-items, filter category equals sample, order by authored fixture sequence and version toy-v1. These labels define only this exercise. They are not actual resource paths, account permissions or stable-snapshot promises. The fixture never changes while the local check runs. That stability makes the demonstration reproducible; it does not establish what happens when a real service's records are inserted, updated or deleted between page requests.

Treat the continuation token as an interface value. Copy it exactly as the declared contract requires rather than decoding it to infer a page number or constructing a plausible successor. In the fictional cases, strings such as cedar and maple are lookup keys supplied in the data. Their spelling carries no numerical meaning. A person should not decide that oak must follow maple just because that sounds like a reasonable sequence of labels.

Keep a useful distinction between a scope change and a retry. A new filter may begin a different traversal even when it uses the same endpoint. A retry may repeat one request under the same scope. Record which event occurred so that a downstream count is not assembled from two different populations by accident. This guide does not set a provider's retry policy; the existing retry-effects resource addresses that separate question. Here the focus is identifying which collection and continuation each page belongs to.

Declare the toy contract and its limits

Every fictional response in this exercise is a JSON object with an items list and a required next field. Each item has a nonempty string id. A nonempty string next value names the next fixture page. An explicit null next value marks the end of this toy chain. A missing next field is malformed under this invented contract. These choices are authored teaching rules, not a claim about the JSON standard, Stripe, Google or a production API's interpretation of missing and null values.

The initial fixture page is named start. Traversal collects item occurrences in their supplied order and follows the supplied next key. It preserves repeated identifiers rather than deciding whether to merge them. If a page is malformed, the traversal stops as malformed and leaves the accepted prior rows partial. If a next key points back to an already visited page, it stops as repeated_cursor and remains incomplete. A safety limit would be another boundary in a real implementation, but this finite fixture does not test an unbounded service.

The expected results distinguish accepted page count, collected occurrences, distinct identifiers, terminal status and completeness. Those are different properties. A repeated identifier can make the distinct count smaller than the occurrence count without proving which occurrence is correct. A short page can still have a continuation. A parser can successfully read a response that does not satisfy the toy contract. Naming those outcomes prevents one general success label from hiding the reason a collection is incomplete.

The JSON fixture contains the page bodies and manually specified expectations. A separate CSV summarizes the cases for reading alongside the article. The local check operates only on the supplied data and compares its results with those expectations. No executable utility is distributed. The code used for editorial checking is not an integration, a complete API client, a security assessment or an authorization decision. The published artifacts are teaching inputs and expected results, not evidence of model performance.

Follow a normal chain without guessing its length

Case A has three fictional pages. Start contains R1 and R2 and points to cedar. Cedar contains R3 and points to maple. Maple contains R4 and R5 and has an explicit null next value. The expected traversal accepts three pages, collects five occurrences and sees five distinct identifiers. Its terminal status is terminal and its complete flag is true for this toy chain. Those words describe reaching the declared end of the supplied fixture, not completeness of any real external dataset.

The middle page is shorter than the first. That fact does not change the authored continuation. A loop that stops whenever a page contains fewer items than the first would incorrectly end at R3. The defect is not in counting the three collected rows. It is in using an unstated stopping rule instead of the response's continuation signal. The example gives a concrete way to ask an assistant to explain why its loop continues even though the second page is small.

Keep the order of occurrences visible in the expected result. R1, R2, R3, R4, R5 is the authored order for A. Sorting the final list by a field before comparing it could conceal a traversal-order difference. Whether order matters depends on the task, but the exercise should not discard it accidentally. A collection used only for a set comparison and one used to preserve a provider's ordering need different checks and explanations.

Do not infer the number of pages from a reported total unless the contract supports that inference. A service may describe a total as an estimate, a different scope or a value that changes during traversal. This fixture deliberately avoids a provider total and follows the chain itself. A real integration can retain any documented total as additional evidence while keeping its meaning separate from the number of rows actually retrieved and accepted.

Continue through short and empty pages when the contract says to

Case B starts with one item, R1, and a next value of birch. Birch contains R2 and R3 and terminates with null. The expected result is three occurrences across two pages. The first page is not an end marker under the toy rules, even though it contains only one item. The case is useful when reviewing a proposed shortcut that treats a small response as evidence that the collection must be exhausted.

Case C starts with an empty items list and a next value of elm. Elm contains R7 and terminates with null. Its expected result accepts two pages and collects one occurrence. The empty first page is still a valid page under this invented contract. A loop that exits whenever items is empty loses R7. This is a boundary example, not an assertion that every API can or will return empty intermediate pages. Check the actual interface rather than generalizing from either assumption.

A useful loop specification separates page validation from item processing and continuation handling. Validating the first page in C does not require there to be an item. Processing contributes zero occurrences. Continuation handling then sees elm and proceeds. Collapsing those steps into if rows, continue can unintentionally make the presence of data control the entire traversal. An assistant's explanation should make clear which condition governs each step rather than rely on a vague statement that it handles empty responses.

A downstream summary should retain these distinctions too. Zero items on one accepted page is not the same as zero items in the complete requested collection. If a process stops after the empty first page, its result should say incomplete and explain the stop. A truthful partial handoff can still be useful for troubleshooting. It becomes misleading when a neat zero count is presented as a finding about the entire source population.

Keep repeated identifiers separate from completion

Case D starts with R1 and R2 and points to ash. Ash contains R2 and R3 and terminates with null. The authored expected occurrence sequence is R1, R2, R2, R3. There are four occurrences and three distinct identifiers, and the chain reaches its declared end. The duplicate identifier does not prevent terminal traversal, but it creates a separate question about how the intended application should interpret those occurrences.

The exercise intentionally preserves both R2 occurrences. It does not silently select the first, the last or a supposedly newer version. In a real interface, repeated identities might arise for different reasons, and resolving them can require fields, ordering guarantees or a provider explanation that the list alone does not supply. A proposed deduplication rule belongs in the transformation specification. It should not be smuggled into pagination merely to make the final count look cleaner.

If the next step is an import, separate retrieval from applying records to a destination. A traversal can collect four occurrences without authorizing four writes or establishing three final destination records. Identity matching, update semantics, missing fields and side effects need their own rules. The fixture performs none of those operations. Keeping the boundary explicit helps prevent an assistant from interpreting finished reading as permission to create, update or delete operational data.

Use both counts in a handoff when both are relevant: four retrieved occurrences and three distinct identifiers in this fictional result. Avoid calling either count the number of customers, tasks or invoices unless that meaning is actually established. The generic id field identifies only the invented items. An exact count can still be semantically wrong when its label claims a population or business entity that the source did not define.

Stop a loop or malformed page without declaring completion

Case E begins with R1 and points to loop. The loop page contains R2 and points back to loop. The expected accepted pages are start and loop, with two occurrences collected before the repeated continuation is detected. The terminal status is repeated_cursor and complete is false. This behavior is an authored safety boundary for the local exercise. It is not a documented recovery policy for a real provider, and it does not establish why a live service might repeat a token.

A repeated token should not be fixed by inventing the next token or discarding the signal and claiming the list is finished. Preserve the request context and report the observed condition in a real investigation. The appropriate retry, restart or provider-contact decision depends on the actual contract and circumstances. This guide's fixture does not simulate network errors, token expiration, authentication changes or rate limits. Its finding is simply that the supplied chain fails to reach the declared terminal state.

Case F begins with R1 and points to broken. The broken page contains R2 but omits the required next field. Under the toy rules, only start is accepted; R2 from the malformed page is not added. The result has one occurrence, terminal status malformed and complete false. Missing next is deliberately different from explicit null here. That distinction comes from this exercise's contract, not a universal rule for every JSON response.

Validate a page before adding its items when the specification requires that behavior. Otherwise, a process may count rows from a response it later rejects and leave the recipient unsure which rows were accepted. Another application could define a different partial-acceptance policy, but it would need to state it. In this exercise, the expected occurrence list is enough to show that the malformed page's R2 never joined the accepted collection.

Give an assistant a precise traversal and handoff brief

An original teaching brief could say: traverse the supplied toy-v1 fixture from start; validate each page against the declared items and next rules; preserve occurrence order and repeated ids; follow each nonempty next key exactly; stop on explicit null as complete; stop on a repeated key or malformed page as incomplete; report accepted pages, occurrences, unique ids and status. This is an authored prompt example, not a model run or an instruction to access a real account.

Ask for a trace alongside the result. A concise trace can list the page key, validation outcome, number of accepted items and continuation decision. That trace makes an incorrect shortcut easier to find without requiring the reader to infer control flow from the final table. It does not need to include secrets or private payloads. For a real workflow, determine what diagnostic information may be retained through the appropriate data-handling rules rather than copying production responses into an unrestricted log.

Before using a real client, define the incomplete handoff. It might contain the accepted partial rows, scope, last accepted continuation and reason for stopping, with a clear statement that traversal is incomplete. Do not let an AI summary omit that flag to produce a smoother report. A table of valid partial records can support a bounded diagnostic task while remaining unsuitable for a completeness-dependent analysis. The recipient needs to know which conclusion the retrieval actually supports.

The downloaded examples reproduce only the authored fixture behavior. They do not measure model accuracy, provider reliability, snapshot consistency or full client conformance. If you later evaluate a generated implementation, retain its actual code or response and the checks used against it. Keep that evidence separate from these manually specified expectations. A successful result on six static chains is a useful narrow result, not a general approval to run an unattended import against an operational system.

Questions about paginated AI-assisted imports

Can a short page tell us the list is finished?

Only when the particular interface defines that as its stopping rule. In the toy fixture, continuation is controlled by next, so cases A and B continue after short pages. Read the endpoint's documented end signal and preserve it in the implementation brief. A convenient heuristic is not a substitute for the contract, even if it works on the first few responses inspected.

Should an empty page always stop the traversal?

No universal rule applies. Case C explicitly permits an empty page with a continuation and therefore retrieves R7 from a later page. Another interface may have a different contract. Validate the response, process its items and interpret continuation according to the applicable rules. If the process stops for a reason other than the declared end condition, its handoff should explain the incomplete state.

Does removing duplicate ids prove the import is correct?

No. Case D has four occurrences and three distinct ids, but the fixture does not define destination identity or update behavior. Removing a repeated id is a transformation decision that needs its own rule and evidence. It also cannot recover records missed by an incomplete traversal. Check retrieval coverage, identity meaning and destination effects separately before drawing a conclusion about an actual import.

What should happen when a cursor repeats?

The toy fixture stops as repeated_cursor and remains incomplete. It does not guess a successor or reinterpret the loop as successful exhaustion. A real provider's recovery behavior must come from its contract and the actual failure context. Preserve enough permitted diagnostic information to understand the condition, then choose the appropriate next step. These examples do not establish a universal retry or restart policy.

Is a missing continuation field the same as null?

That depends on the interface. This invented toy contract requires next, and only explicit null signals its end. Case F is malformed because the field is absent. Other APIs may document omission as their terminal signal. An assistant should retain the rule that applies to the actual endpoint rather than promote the teaching fixture's choice into a general JSON or API standard.

What does complete mean in the downloadable expectations?

It means the local traversal reached the declared terminal signal of the supplied static chain while following the toy rules. It does not mean a live database snapshot was captured, every business entity was retrieved or any destination writes are authorized. The files contain original fictional inputs and manual expectations, not customer records or observed model outputs. Use them to clarify a traversal specification and expose narrow boundary errors.