REST API Pagination Patterns for Large Web Dataset Retrieval

Pagination decides whether a data pipeline survives contact with a real dataset. Pick the wrong pattern and an API that works fine in testing falls over at page 5,000 in production, or quietly drops records nobody notices until a RAG index starts returning garbage answers. This piece breaks down the four major REST pagination patterns, where each one holds up, and where each one collapses once an AI pipeline starts pulling web data at real scale. One conclusion up front. Offset pagination, the pattern almost everyone learns first, is the wrong default for this work, and treating it as a safe starting point is how teams end up debugging silent data loss six months later.

Most pagination tutorials assume a dataset that barely changes and a client that only needs to page through a few dozen results. Web retrieval for AI pipelines is the opposite situation. High volume, constant change underneath the pipeline while it's mid-request, and downstream consumers (vector indexes, agents, monitoring systems) that fail silently when a batch comes back incomplete or stale. Two failure modes hide in that gap. One is performance collapse at depth, where a query that returns in 20 milliseconds on page 5 takes seconds on page 5,000, and nobody notices until the pipeline times out under load. The other is data drift, where records get skipped or duplicated because the dataset shifted between one page request and the next.

Skip pagination entirely and a dataset of even modest size floods server memory and chokes on timeouts. Pick the wrong pattern and the result is the same failure, just slower and quieter, which is worse. An agent that missed 40 records on page 300 has no idea it missed anything. So pagination itself was never really the question. What matters is which pattern survives the actual conditions the pipeline runs under, and for most AI retrieval work at scale, that rules out offset before the conversation even starts.

How offset pagination works and where it stops working

Offset pagination is the one everybody learns first, because it maps directly onto SQL. Two parameters do the work: limit sets how many records come back, offset sets how many to skip. A request like GET /api/items?offset=20&limit=10 becomes SELECT * FROM items LIMIT 10 OFFSET 20. Simple, stateless, and it lets a client jump straight to page 47 without walking through pages 1 through 46 first. For a small admin dashboard, that's genuinely enough.

The trouble starts with depth. OFFSET 99980 doesn't teleport the database to row 99,981. It scans from row 1, counts off the first 99,980 rows, and throws them away before returning anything. Cost scales linearly with page depth, so page 5 and page 50,000 stop being the same kind of query, even though the code looks identical.

Then there's the correctness problem, and this one is worse because it fails silently. Delete a row between two page requests and every row after it shifts position by one. A record sitting at position 21 slides into position 20, lands inside the window page 1 already returned, and vanishes from page 2 entirely. No error, no warning. The client just never sees it. For a manual admin panel, that's a minor annoyance. For an automated ingestion pipeline feeding a search index, it's a silent, permanent gap nobody can trace back to a cause.

Offset does have a narrow place, and it's worth being honest about how narrow it actually is. Enforce a maximum page size (50 or 100 records is typical), reject or clamp anything larger, and default missing parameters to page 1 with a sane limit like 10 or 20. That's the ceiling of what it's good for: small static datasets, UIs that need numbered page links or a "jump to page N" control, and anything where write volume stays low enough that rows aren't shifting mid-traversal. Past that, it's the wrong tool, full stop, and no amount of tuning fixes what's structurally broken about it.

Cursor and keyset pagination: how indexed seeks replace row scanning

Keyset pagination flips the mental model entirely, and it's the pattern that should be the default for anything built after 2020. Offset gives an address by row number, fragile because that number changes every time something gets inserted or deleted upstream. Keyset gives an address by content: a value that stays fixed no matter what happens to the rows around it.

Instead of asking the database to skip N rows, the query asks for rows where the ID is greater than the last one seen: GET /api/items?after_id=100&limit=10 becomes SELECT * FROM items WHERE id > 100 ORDER BY id ASC LIMIT 10. Because the ID column is an indexed primary key, the database performs a B-tree seek straight to that point, and it doesn't scan anything that came before it.

That's what makes the performance profile flat. Finding row 50 and finding row 5,000,000 cost roughly the same, because the database is seeking to a value, not counting rows to discard. Practitioner testing on this pattern found page 1,000 loading in 45 milliseconds under cursor pagination, 177 times faster than the same query run with offset. And because position is anchored to a value instead of a row count, inserts and deletes elsewhere in the table don't shift the anchor. No silent skips, no duplicate records on the next page.

Sorting on more than one column needs a compound cursor. In a relational database, that looks like a query filtering on the last seen sort-key and ID combination, ordered by those same fields ascending, limited to a fixed page size. The compound key keeps positioning unique even when two records share the exact same timestamp.

One condition matters more than any other here: the columns in that WHERE clause need an index that actually supports the comparison. Skip that step and the query degrades into a full table scan on every page request, worse than offset, since now it's paying scan cost plus the overhead of evaluating the boundary condition. Check with EXPLAIN ANALYZE that the plan shows a seek, not a scan. Anyone shipping keyset pagination without running that check is shipping a guess dressed up as an optimization.

The trade-off is real: keyset pagination can't tell a client how many total pages exist, and it can't jump to page 40 out of nowhere. For a web-data pipeline, that's rarely worth losing sleep over. For a UI that needs numbered navigation, it rules the pattern out entirely. Past roughly 10,000 records, cursor-based pagination stops being a nice-to-have and becomes the practical default; Moldstud reported the response-speed improvement from cursors at up to 50% over traditional offset pagination.

Time-based and delta pagination for continuously updated web sources

Offset and keyset both assume the goal is reading a dataset in full. A lot of web-data work doesn't need that. It only needs what changed since the last time the pipeline looked, which calls for a completely different shape of query.

The mechanism adds a since or updated_after parameter that anchors the request to a timestamp instead of a row position. A request anchored to a timestamp can be combined with a limit parameter, so a burst of updates within one window doesn't overwhelm a single response. Because only records newer than the anchor even get considered, this sidesteps the full historical scan pure offset pagination is stuck doing. Pairing it with HTTP headers like If-Modified-Since and Cache-Control cuts load further when the source hasn't changed at all.

The timestamp column needs an index, same requirement as keyset, or the query collapses to a full scan just like offset would. Time-based pagination carries its own specific failure mode too: sparse windows return empty pages, and time zones introduce real correctness risk if the pipeline doesn't normalize everything to UTC and account for clock skew between the source system and whatever's consuming it.

Where it earns its place is change detection: re-crawl scheduling, monitoring feeds for updates, refreshing a RAG index. A RAG index built last month carries two separate problems. Pages that didn't exist yet are a coverage gap, that one's obvious. Pages that changed since the last crawl but still have old embeddings pointing at old text, that's drift, and it's the more dangerous of the two because nothing about the query looks wrong. Time-based pagination goes straight at the drift problem, because it makes incremental re-fetching cheap enough to run often.

Token-based and opaque pagination for stateful API traversal

Token-based pagination hands the client a next_page_token (sometimes next_cursor) with every response, and the client's only job is to pass it back verbatim on the next request. No decoding, no guessing what's inside it.

All the state, sort order, filters, current position, is encoded inside that token, opaque to the client. It might be a base64-encoded cursor or a signed offset bundling position and filter context. From the client's seat, this is strictly linear: follow the token forward until the response stops sending one, and that's the end of the dataset. No jumping ahead, no computing anything.

Some APIs go a step further with HATEOAS, returning actual hypermedia links, next and prev, directly in the response body. The client just follows the link instead of building the next request by hand.

The failure mode that matters for automated pipelines is expiry. Tokens are often time-limited, and a pipeline that pauses mid-traversal, whether from rate limiting, backpressure, or a transient error, can come back to find the token dead with no way to resume from where it left off. Any pipeline built on a token-paginated API needs to checkpoint completed pages to durable storage as it goes, so a restart picks up close to where it stopped instead of replaying the entire dataset from page one. If a third-party API response includes a field called next_page_token or continuation_token, treat it as a black box. Never try to parse it or reconstruct it manually.

How each pattern behaves under the specific conditions of AI pipeline retrieval

Akamai tracked a steady climb in LLM scraper activity between March and April 2025, with traffic from major model providers, OpenAI, Meta, Anthropic, Google, contributing to that growth. That volume is exactly the condition under which pagination weaknesses stop being theoretical and start showing up in production.

Four questions decide which pattern actually fits a given pipeline. How deep does retrieval need to go? Offset degrades with depth; keyset doesn't. How volatile is the source between requests? Offset creates gaps as records shift, while time-based delta keeps re-fetch cost low by only asking for what's new. Can the pipeline pause and resume cleanly? Token-based pagination demands checkpointing to survive that, while keyset cursors resume from the last seen value without any extra bookkeeping. And how fresh does the downstream index need to be? The tighter that requirement, the more the answer points toward time-based delta or a continuous crawl trigger instead of a full re-pull.

This matters most for RAG. Classic retrieval-augmented generation was built with a slow-moving knowledge base in mind. Pointed at the open web instead, a vector index built a week ago confidently returns a week-old version of reality, with no signal that it's out of date. The pagination pattern feeding that index determines how wide or narrow that staleness window actually gets.

There's a maintenance cost underneath all of this too. Kadoa found that roughly 80% of engineering time on traditional scraping architectures goes into maintaining scrapers rather than building anything on top of them. Pagination bugs contribute to that burden, because a broken cursor doesn't throw an error. It just quietly returns less data than it should, and nobody finds out until the output downstream looks wrong for reasons that take days to trace back to the actual cause.

Laid out plainly: offset belongs in small, static, low-write datasets and admin UIs, not production AI ingestion at any real scale. Keyset and cursor pagination fit large or fast-growing datasets that need stable performance and clean resumability, and it's the pattern most teams should reach for by default. Time-based delta fits continuously updated sources and RAG refresh cycles, anywhere the goal is catching only what changed. Token-based pagination shows up when consuming someone else's API that manages state server-side, and it demands a checkpoint strategy to survive expiry.

Structuring paginated responses so downstream systems can consume them reliably

The pattern chosen doesn't just shape the query. It shapes every response the pipeline has to parse afterward, and an inconsistent response shape turns into its own maintenance headache over time.

A well-built paginated response gives the consumer a handful of things it actually needs. A position anchor, whether that's a cursor value, a token, or an offset, so the client can checkpoint exactly where it stopped. A next link or next_page_token field, so the pipeline never has to reconstruct the next request from scratch by guessing at parameters. And a clear signal for whether more pages exist, either a boolean flag or the simple absence of a next field, so the pipeline knows traversal is done without wasting a request to find out.

Total count fields are useful for offset-driven UIs but expensive to compute at scale, since they usually require a full COUNT(*) query behind the scenes. Cursor-based and token-based APIs can skip that field entirely, since most downstream consumers never need it anyway.

HATEOAS-style links (self, next, prev) in the response body make an API self-describing, which matters more than it sounds like once multiple pipeline implementations are all consuming the same endpoint. Error handling needs the same discipline. If one page request fails, the pipeline should retry that exact page, and that only works if the position anchor is stable enough to be held externally. Keyset and token patterns support that. Offset does not, once the underlying dataset has shifted underneath it.

Rate limit headers, X-RateLimit-Remaining and Retry-After, belong in the response contract too, so a pipeline backs off on its own instead of hammering an endpoint until it gets blocked outright. For web-specific retrieval, including a content hash or a Last-Modified timestamp on each record lets a downstream RAG indexer skip re-embedding content that hasn't actually changed, saving real compute without giving up any freshness.

What a well-designed retrieval API looks like when pagination is handled at the infrastructure layer

Teams that build and maintain their own crawlers and paginators end up spending most of their time keeping that machinery running rather than building anything with it. Kadoa's 2026 figure, around 80% of engineering time going to maintenance, largely traces back to site redesigns, selectors breaking without warning, and anti-bot defenses shifting underneath a scraper that worked fine last month.

A retrieval layer that handles pagination correctly removes a specific list of burdens from the application developer. Cursor management and checkpointing, so nobody's hand-rolling resume logic. Index verification, confirming keyset queries are actually seeking and not scanning. Token expiry and restart handling for third-party APIs that manage state server-side. Delta detection for time-based refresh, so re-crawls only pull what changed. And separately from all of that, there's script-rendered content sitting behind "Load More" buttons, a pagination problem in practice even though none of the four patterns above solve it at the plain HTTP level.

The right abstraction folds all of that into a single call: one REST endpoint that takes a URL or a defined crawl scope and hands back clean, structured data, with the pagination pattern, the indexing, the checkpointing, and the retry logic already handled underneath it. Whichever pattern feeds a pipeline, offset, keyset, time-based delta, token-based, the pipeline itself should never have to know or care.