Full-Site Crawl vs Targeted URL Fetch for AI Data Pipelines

Full-site crawl and targeted URL fetch solve two different problems, and mixing them up costs real money. Cloudflare's own bot data shows model training now drives nearly 80% of AI bot activity. Crawl decisions that used to belong to SEO teams now sit on the desk of AI infrastructure teams instead. A Censuswide survey of web scraping professionals found 74% reporting more demand for web data over the prior 12 months. Most teams still pick a mode out of habit, not fit, and that habit is expensive: wasted tokens, added latency, engineering hours spent building infrastructure nobody needed. Crawl is for when you don't know what's out there. Fetch is for when you do. Picking the wrong one is the single most common architecture mistake in this space, and it's an easy one to avoid once you know what each mode is actually built to do.

What full-site crawl and targeted URL fetch do, precisely defined

A full-site crawl starts from a handful of seed URLs, pulls the links off each page, queues the new ones, and comes back later to check whether pages it already visited have changed. The output is a map: what exists on a domain, and what's different since last time.

Underneath that, a crawler does four separate jobs. It discovers new links. It revisits known URLs on some schedule. It fetches the raw content. Then it hands that content off to whatever's downstream, whether that's a parser, a search index, or an embedding pipeline.

Targeted URL fetch is narrower, and it should stay narrow. Someone already knows the page they want. The work is retrieval and cleanup, nothing more: pull the URL for a product page, strip the junk, hand back something usable. No link graph, no revisit schedule, no exploration. Teams that bolt crawl features onto a fetch job are usually solving a problem they don't have yet.

A third mode has crept into production systems: hybrid or incremental crawling. Instead of re-crawling an entire domain every cycle, the system checks which pages actually changed and only re-indexes those. For large sites on a re-index schedule, this is often the only mode that makes financial sense. Teams that default to full re-crawls on every cycle are burning budget for no reason, full stop.

One gap deserves more attention than it usually gets. Per data from Vercel and MERJ, none of the major AI crawlers, not GPTBot, not ClaudeBot, not PerplexityBot, render JavaScript. They pull raw HTML and stop. Google's Gemini is the exception, since it rides on Googlebot's shared rendering infrastructure. So if a site injects its main content client-side, most AI crawlers never see it. That has to be a design-time decision, made before a RAG index comes back half-empty, not something discovered after the fact.

Size is a limit too, and not a small one. Googlebot caps its fetch at 2 MB for supported file types. The median mobile homepage in 2025 runs around 2,362 KB. Even the most well-resourced crawler on the planet is bumping against a ceiling that quietly decides what it actually sees.

When full-site crawl is the right primitive

Crawl wins when the URL inventory itself is the unknown. Documentation sites, wikis, e-commerce catalogs: anywhere the full set of pages isn't already sitting in a spreadsheet. A single crawl call can walk an entire docs site and come back with everything, which beats listing pages out by hand one at a time.

It also wins for anything needing domain-wide coverage rather than a handful of pages. Building a RAG index off a documentation site is the clean example: a crawl preserves the heading hierarchy across every subpage it touches, and that hierarchy is what the chunking step needs later. Fetch a bunch of individual pages one at a time and that cross-page structure just isn't there, because nothing connected them in the first place.

Change detection is a native crawl behavior. Revisiting known URLs on a schedule to catch what's different is the crawler doing its actual job. A single fetch call has no memory of what the page looked like last week, and it isn't supposed to.

Crawl has a quieter use too: mapping a site before extraction. Pull a sitemap or hit a map endpoint first, isolate the URLs actually worth extracting, then run targeted fetches against that shortlist instead of crawling blind. That sequencing alone saves a lot of wasted crawl budget. Skipping it is one of the more expensive habits teams fall into, and it's avoidable with about ten minutes of planning.

None of this comes free. Full crawls at scale get expensive fast, so scope them on purpose rather than defaulting to "crawl everything and sort it out later." Crawl also isn't built for pulling one specific field, a price, a stock count, off a page already known. That's fetch's job. Forcing crawl to do fetch's work is where most of the waste mentioned earlier actually comes from.

The scale at which production systems run crawl mode is striking. Anthropic executed roughly 38,000 crawls per referred visit in July 2025. That's continuous infrastructure running at a volume most engineering teams don't plan for until they've already blown through a budget.

When targeted URL fetch is the right primitive

Fetch wins the moment the URLs are already known. Price monitoring, competitor tracking, a fixed set of landing pages: anywhere the destination is decided and the only open question is what's on it right now.

It's also the right shape for agent tool-use. In a connected-agent loop, search finds candidates, fetch pulls the page, the model reads it, then answers. That fetch step needs to be fast and clean. Dragging full-crawl overhead into step three of that loop is the wrong tool in the wrong slot, and it appears as latency the person on the other end can feel.

Structured field extraction from a known page, pulling a price, a stock flag, a specific table, fits fetch too, especially when the goal is JSON back rather than a wall of text to parse again downstream. And for RAG systems already built, fetch is what handles the refresh: re-index the one page that changed instead of re-crawling the whole domain to catch it.

Output format is where the fetch layer either saves money or quietly burns it. Raw HTML full of nav bars, ad slots, and script tags creates real downstream cleanup work. Per Webclaw's RAG pipeline documentation, LLM-ready Markdown or JSON cuts token count by roughly 90% compared to raw HTML, and that difference lands directly on the embedding API bill.

Some fetch tools popularized turning any URL into clean Markdown with almost no setup, genuinely useful for quick prototyping. But tools built that way tend to optimize for single pages rather than site-wide discovery, and that's fine as long as they're used for what they're actually good at instead of stretched past it.

Latency matters here in a way it doesn't for crawl. Sub-300ms response times are achievable on static pages with well-designed tooling. In an interactive agent loop where a person is sitting there waiting on an answer, that speed is the whole point, not a bonus feature.

Cost math splits here too, so do the arithmetic before committing to a method. CSS selector extraction, once built, costs close to nothing per page. LLM-powered extraction costs tokens on every single page it touches, and since fetch mode runs per-URL, that cost compounds fast across a large list. Know which extraction method a job actually needs before turning an LLM loose on ten thousand pages. The wrong choice there is expensive in a way that's hard to walk back.

Output format's effect on pipeline costs and quality

Getting the page isn't the finish line. A tool can fetch successfully and still hand back raw HTML that needs a pile of post-processing before anything downstream can use it. That gap between "retrieved" and "usable" is where a lot of pipeline cost quietly hides, and it's the part teams underbudget for most.

Clean Markdown or well-formed JSON is easier to chunk, store, and pass to a model. It isn't a nicety, it's a line item. Per RAG pipeline documentation from Webclaw, moving to LLM-optimized formats cuts token count by roughly 90% against raw HTML. At current embedding pricing, that percentage often decides whether a recurring ingestion job is worth running at all.

Strip the heading levels out during scraping, and the chunking step loses the document structure the retriever leans on to figure out what belongs with what. That loss doesn't appear as an error message. It occurs later, as bad retrieval that nobody can trace back to its source.

One finding here should reset how people think about schema markup, and it's a stronger claim than most teams are ready for. Research from SearchVIU found that pages storing data only in JSON-LD, Microdata, or RDFa saw near-zero extraction success from major AI crawlers. Claude pulled zero prices out of any schema format tested. ChatGPT managed a 37.5% success rate, and even that came primarily from visible HTML rather than the structured markup.

Structured data in the page source is not doing the heavy lifting most teams assume it is. It is not a substitute for clean visible HTML when an AI system is the one reading it, and treating it as one is a bet the SearchVIU numbers say doesn't pay off. Design around what the crawler actually parses.

And boilerplate, nav bars, cookie banners, ad units, that cleanup belongs at the API layer. Every team that builds its own boilerplate stripper by hand is rebuilding infrastructure that should have shipped already solved.

Structured extraction via JSON schema in the mode decision

Schema-driven extraction flips the usual scraping problem around. Hand the tool a JSON schema, get back data shaped to match it, and skip the CSS selector that snaps the moment a page layout shifts.

What actually matters is whether the schema is enforced at the token level, or just suggested to the model as a request it might follow. That distinction, more than anything else in this section, decides whether the system holds up under production load or starts silently dropping fields nobody notices until a report comes back wrong.

A preprint on a system called PARSE looked at autonomously optimizing JSON schemas for LLM consumption and found up to a 64.7% improvement in extraction accuracy on the SWDE benchmark, with combined framework gains reaching 10% across models tested, and a 92% drop in extraction errors within the first retry. Schema design itself is a lever teams should keep pulling on.

Schema extraction fits naturally with targeted fetch, since the URL and the fields wanted are both already known. But it also runs cleanly over crawl output: crawl finds the URLs, then extraction runs against each one, often in parallel.

The cost split from earlier appears again here. LLM-powered schema extraction costs tokens per page. CSS selectors cost almost nothing once written, but they break the moment the page changes. Which one makes sense depends on how often the target pages actually change layout, and how much selector upkeep a team can stomach before it becomes its own maintenance job.

The field splits along this same line. Some tools hand back model-ready JSON on the first call. Others hand back raw HTML and leave every bit of structuring work to whoever called them, which is a cost the caller pays whether or not it appears on an invoice.

How agentic pipelines change which mode you reach for

Inside an agent loop, search finds candidates, fetch retrieves the page, the model reads it, and an answer comes out the other end. That fetch step has to be quick, dependable, and clean on output. Full-crawl overhead doesn't belong in that loop, and forcing it in there is a common mistake dressed up as thoroughness.

MCP support has become a real selection criterion for scraping APIs used inside agent pipelines. Skip it, and every new agent framework the team adopts means writing fresh glue code to bridge the gap, over and over, for a problem that's already been solved elsewhere.

There's a useful scale heuristic floating around production teams: for anything under roughly 100 data points, a coding agent or browser agent can often just handle it without dedicated infrastructure. Past that, for recurring production jobs, the deciding factors shift to concurrency limits, queueing, retry logic, and rate-limit handling, the boring plumbing that only matters once volume raises the stakes.

The A2A protocol, donated to the Linux Foundation with 150+ supporting organizations, adds gRPC support and defines how agents hand off sub-tasks to each other. In multi-agent setups, the agent responsible for fetching data is often a dedicated sub-agent, which strengthens the case for a purpose-built fetch primitive over ad-hoc HTTP calls that don't compose the same way inside a system built around delegation.

Most failures in AI scraping systems don't happen at the parsing step anyway. They happen upstream, at fetch: the page never came back in the first place, blocked by anti-bot defenses, missed because of the JavaScript rendering gap, or throttled by a rate limit. Fetch reliability, not parsing sophistication, is the thing worth screening for in agent tool-use.

A practical decision framework: matching mode to pipeline job

Four questions settle most of these decisions, and skipping any one of them is how teams end up rebuilding pipelines six months in.

Is the URL inventory known or not? Unknown inventory means crawl first, build the map, then extract. Known URLs mean skip the crawl step entirely and fetch directly, no exceptions.

Is the goal coverage or precision? Comprehensive domain coverage, a RAG index, a site audit, ongoing change monitoring, that's a crawl job. Specific fields off specific pages, a price, a status flag, a named data point, that's targeted fetch paired with a JSON schema.

How much latency can the pipeline tolerate? An interactive agent loop with a person waiting needs fetch only, tuned for low latency. A batch ingestion job running overnight on a schedule can tolerate crawl, prioritizing throughput and freshness over speed.

How often does the content actually change? Fast-changing content that needs frequent re-indexing calls for incremental or hybrid mode: detect what changed, re-fetch only that. A stable documentation corpus can sit on a slower full-crawl schedule without losing anything.

Token budget is the constraint running quietly under all four. A full crawl across a large domain produces a lot of content, and if only a slice of it is actually relevant to the pipeline, a well-scoped fetch list ends up cheaper per useful token than crawling everything and filtering after the fact.

The strongest architectures don't pick one mode and stop there, and treating this as an either/or choice is where a lot of teams go wrong. They run a crawl to discover and map a domain, then dispatch targeted fetches to do the actual extraction. Treat the two modes as sequential steps in one pipeline.

Key features of an API that handles both modes cleanly

Clean Markdown or JSON at the API layer, with boilerplate stripping and format normalization already solved by the provider. If a team is writing its own HTML cleanup step in 2025, that's a sign the tool is the wrong one.

JavaScript rendering that's available on request but not forced on every call. Teams need headless rendering for JS-heavy pages, without paying that rendering cost on every static page that never needed it.

Schema-based extraction that actually accepts a caller-supplied JSON schema and returns data conforming to it, rather than pushing the caller back toward hand-written CSS selectors or manual LLM output cleanup.

An MCP server is close to table stakes at this point for anything touching agent pipelines. Without one, every new agent framework adopted means another round of custom glue code that shouldn't need to exist.

Anti-bot handling has to be built in from the start. Fetch success rate is the metric that actually shows whether a pipeline works, and proxy rotation plus block bypass should be the provider's job, not something the caller engineers from scratch on a deadline.

Sitemap and URL discovery endpoints let a team check URL inventory before committing to a full crawl. A crawl API that exposes a map endpoint turns that check into real cost control.

Scalability under concurrent load separates tools that survive a demo from tools that survive production. Queueing, retries, and rate-limit handling need to be the platform's job, because a tool that handles five pages fine and falls over at five thousand fails the only test that actually matters for recurring ingestion.

SDK coverage across the languages a team actually uses, TypeScript, Python, Go, Ruby, PHP, decides how fast integration goes. Going from a cold start to working API calls should take minutes.

How leading tools handle the crawl-vs-fetch spectrum in practice

The pattern across serious scraping infrastructure today is specialization by API endpoint. An HTML API handles full headless browser rendering for JS-heavy pages, returning the complete DOM after scripts have run. A sitemap API handles discovery separately, parsing a site's sitemap to build a full URL inventory before any crawl commits resources to actually visiting those pages.

That separation is the shape the whole industry is converging on: discovery as one step, rendering as another, extraction as a third. It maps directly onto the decision framework above. Know what exists, decide what's worth fetching, then extract what the pipeline needs and nothing more.

Pay attention to the tools that don't claim to do everything in a single call. They're the ones that let an engineer pick the right primitive for each step and get out of the way otherwise, and that restraint is the actual sign of a tool built by people who understand the job.