Handling Pagination and Infinite Scroll in Extraction Pipelines

Pagination is where AI pipelines lose completeness, and they lose it quietly. A crawler that stops at page one of a product listing, a news archive, or a regulatory database hands every downstream system, RAG indexes, brand monitors, agentic tools, a partial picture of the world and calls it done.

No error fires when this happens. A paginator that stops early still returns data, so the pipeline looks healthy, the logs stay clean, and nothing in the system flags that pages two through fifty never got touched. That silence is what makes pagination an architecture problem rather than a scraping detail: the failure doesn't announce itself, it just sits downstream as a gap nobody asked about. For retrieval systems, the damage compounds in a specific way: a RAG index can only surface documents it already holds. And the sites feeding these pipelines keep making this harder. Plain URL parameters are giving way to JavaScript-driven patterns, infinite scroll, load-more buttons, which raises the cost of getting pagination wrong at exactly the moment more pipelines depend on getting it right.

The five pagination patterns and their structural differences

Five patterns cover nearly everything a crawler encounters, and they don't just look different on the surface. Each one makes a different promise, or no promise at all, about consistency, order, and whether you can trust that you've seen everything. Picking the right handling strategy starts with knowing which promise you're actually working with.

| Pattern | How it signals "next" | What makes it distinct | |---|---|---| | Offset / page-number (?page=3, ?offset=40) | URL math | Easy to detect and traverse, but no protection against the data shifting underneath you | | Cursor-based (?cursor=eyJpZCI6..., after_id=12345) | A bookmark from the response | Stable under concurrent writes because it tracks position, not an index | | Next-link discovery (rel="next", embedded next-URL tokens) | Parsing the page itself | No arithmetic involved, and it terminates naturally when no next-link shows up | | Infinite scroll | Scroll-triggered AJAX/XHR calls | No button to find; requires triggering a JavaScript event to see the request at all | | Load-more buttons | Click-triggered AJAX/XHR calls | Same underlying mechanics as infinite scroll, different trigger |

Offset pagination is the simplest to spot. The page number or offset sits right in the URL, so traversal is just counting. That simplicity is also the trap, which the next section covers in full.

Cursor-based pagination works differently: instead of counting pages, it carries a bookmark forward, often a primary key or timestamp, that stays valid no matter what else happens to the dataset while the crawl runs. The catch with cursors is that the next token lives in the response, not in a formula, so the extraction logic has to read each page before it can ask for the next one.

Next-link discovery pushes that same idea further: there's no math at all, just a parser looking for a rel="next" anchor or a next-URL token buried in the HTML or JSON. The sequence ends itself: no next-link means no more pages, full stop. Three habits keep this pattern from producing silent bugs: deduplicate every item by a canonical key, watch for soft-blocks (small HTML bodies that are technically a 200 response but missing the selectors you expect), and log a checkpoint at every step so a failed run can pick back up where it left off.

Infinite scroll is the hardest of the five, because there's no next-link, no page number, nothing to parse until a scroll event fires and the page requests more content on its own. Without a hidden API to call directly, this pattern requires a headless browser like Playwright or Puppeteer, with an explicit comparison of scroll height before and after each scroll to tell real content from a page that's simply stalled. Browse AI treats these as distinct, explicit settings, "Click next," "Load more items," "Infinite scroll," because the interaction needed to advance genuinely differs and has to be written into the robot's instructions.

Why offset pagination fails silently at scale

Offset pagination earns its popularity honestly: it's the easiest pattern to build and the easiest to reason about. That same simplicity is what makes it dangerous, because its failure mode doesn't throw an error. It just quietly gets the data wrong.

The mechanism is simple to describe and easy to miss in practice. No warning accompanies either outcome. The run finishes, the row counts look plausible, and the dataset is wrong in ways nobody notices until much later.

Two mitigations help when offset can't be avoided. Second, scope the crawl to a fixed time window: pull only items created before the run started, so new inserts during the crawl simply fall outside the query and never create the shifting problem in the first place.

Cursor-based pagination is the stronger long-term fix, not just the safer one. A cursor points to a position, not an index, so it holds steady even while the dataset changes underneath it, and it doesn't carry offset pagination's other problem either: deep pages in an offset scheme force the database to scan and discard everything before the requested page, which gets slower as the dataset grows. For AI pipelines, the stakes around the correctness issue run higher than they do for a person doing the same review by hand. A RAG index or an agentic tool-use loop has no equivalent instinct. It just works with whatever it was given.

Handling infinite scroll and load-more without headless browsers when possible

The instinct to reach for a headless browser the moment a page scrolls instead of paginating is usually solving the wrong problem. The scroll event is just a trigger in the interface. The data itself almost always arrives through a plain HTTP call that can be replicated directly, with no browser involved.

Finding that call takes one simple step: open the browser's DevTools Network tab, scroll once, and watch which XHR or Fetch requests fire. Once that hidden API turns up, the problem is no longer about scrolling. It becomes ordinary cursor or offset handling, the same logic already covered, now applied to JSON responses.

Sometimes there's no API to find, and JavaScript execution really is unavoidable. In that case, Playwright or Puppeteer can drive the page, but the detection logic needs to be explicit: compare scroll height before and after each scroll, and treat a delta of zero as the only reliable sign that content has run out. Waiting for the network to go idle holds up far better.

The cost gap between these two approaches is large. Running a headless browser for every JavaScript-heavy page costs far more in compute and time than a plain HTTP fetch, and that difference adds up fast across a crawl spanning thousands of pages. Even then, the same first move still matters: check for a hidden API before assuming browser automation is required, since that choice shapes how the request itself gets built.

Deduplication, stop conditions, and checkpointing as non-optional pipeline hygiene

Every paginated loop needs three things regardless of which pattern it's traversing: a way to know when to stop, a way to avoid counting the same item twice, and a way to pick back up after a failure without starting over. Skipping any one of these turns the loop into a prototype: it can run forever, quietly corrupt its own output, or demand a full restart the next time something breaks.

Stop conditions need to check more than the HTTP status code at every page boundary. If page N returns the same product IDs or content hashes as page N-1, the loop has stalled or started repeating itself, and it needs to stop immediately even though the URL technically changed. And every loop, especially infinite-scroll loops where there's no guaranteed signal that content has ended, needs a hard ceiling on page count or item count as a last-resort stop.

Deduplication has to happen below the URL level. Deduplicating on a stable business key, a product ID, an article ID, a content hash, before the item reaches the downstream index catches what URL comparison alone misses.

Checkpointing covers the failure case: any paginated crawl that runs for minutes or hours needs to resume from the last page it finished, not from page one, after a crash, a rate-limit block, or a dropped connection.

For RAG pipelines specifically, checkpointing protects more than just crawl efficiency. A partial re-crawl that only overwrites the pages it actually reprocessed keeps the index from ending up as a mismatched blend of old and new versions of the same content.

Pagination failures and RAG quality and agentic tool-use reliability

A missing page in a RAG system doesn't produce an error message. It produces a worse answer that looks exactly as confident as a good one. Retrieval can only return candidates from what got indexed, so a gap in the crawl becomes a gap in the pool the model picks from, and nothing in the output signals that something relevant was ever left out.

Freshness makes this worse. Pagination handling sits right at the point where this breaks: a crawler that misses pages two through N of a product listing or a regulatory archive leaves those pages permanently absent from the index, not temporarily delayed, permanently gone until the next full crawl catches them, if it catches them.

Agentic tool-use carries its own version of this risk. The agent has no way to know what it didn't see, so it produces an answer that sounds complete while resting on a fraction of the real picture.

LinkedIn's agentic customer support system shows one way to build around this. Its ETL pipeline enforces snapshot versioning: each refresh supersedes the prior snapshot through a snapshot pointer, and old content isn't deleted, just scoped out of what retrieval can see. That architecture only works if what feeds it is complete. The pagination and crawl layer must capture the full state of the source, or each snapshot will be a partial one that happens to look full.

Choosing between self-managed pagination logic and a managed extraction API

The right choice here comes down to where a team's actual advantage lives. If the product's value sits in what gets done with the data, not in how the data gets collected, then building and maintaining a browser fleet and custom pagination logic is infrastructure work that competes for engineering time against the thing that actually differentiates the product.

Self-managing pagination, with Playwright or Puppeteer for JavaScript-heavy pages, custom cursor logic, proxy rotation, and retry queues, gives full control over the crawl. For teams building LLM-oriented crawlers who want that level of control, Crawl4AI, the open-source project maintained by Unclecode, is the established option in this space. Maintenance doesn't stop once the crawler works, either: bot-detection countermeasures change, browser versions update, and selectors drift, each demanding ongoing attention.

A managed extraction API shifts rendering, proxy rotation, retry logic, and rate-limit handling onto the provider, leaving the team responsible for target permissions, data quality checks, and what happens to the data after it arrives. For the common case, clean data from a defined set of URLs, a managed API usually wins on engineering time saved against infrastructure maintained. Browse AI's no-code approach fits teams without dedicated engineering capacity for this kind of work: train a robot once, pick the pagination mode explicitly (next-button, load-more or infinite scroll, or no pagination at all), and chain robots together through workflows for crawls that need multiple paginated steps.

The decision comes down to two questions. If a site demands complex multi-step interaction, authenticated sessions, or a pagination pattern a managed API doesn't support, self-manage it. If the actual need is clean content from paginated public sources at scale, in a form ready for an LLM to use, a managed API skips weeks of infrastructure work that would otherwise sit between a team and its first working feature.

Encoding pagination strategy as a first-class decision

Pagination handling that works today won't necessarily work next month. Sites change their patterns, page numbers give way to infinite scroll, a public API gets replaced by an authenticated cursor endpoint, and a pipeline with no written record of its pagination strategy has no way to recover when that happens. It just starts quietly missing data again.

Treating pagination as a first-class decision means writing a few things down for every source: not just the URL, but the specific pagination type in use, the cursor field or next-link selector, the stop condition, and the deduplication key. And it means logging a checkpoint, cursor or page number, item count, timestamp, at every page boundary, so a failure costs only minutes of reprocessing.

This matters even more as agentic systems take on more of the retrieval work themselves. Web retrieval infrastructure for LLM agents is moving toward architectures where agents discover and invoke tools dynamically. Pagination logic built correctly into one of those tools means the agent inherits correct traversal without ever having to reason about it. Pagination logic built poorly becomes a capability gap the agent has no way to compensate for, because it can't see what the tool never fetched.

For any new paginated source, the same short list applies every time: identify the pattern, whether it's offset, cursor, next-link, infinite scroll, or load-more. Check the Network tab for a hidden API before committing to browser automation. Choose cursor over offset wherever the site allows it. Put a stop condition, item-hash deduplication, and checkpoint logging in place before the first production run, not after the first incident. And decide whether to self-manage or delegate to a managed API based on how complex the interaction actually is and where the team's real differentiation lives.