Horizontal Scaling Strategies for Distributed Crawlers

A crawler becomes a system rather than a script once you have to split the work across machines. That split forces five separate design problems onto you at once: how URLs get queued, how fetches get distributed, how pages get parsed, how duplicates get caught, and how politeness rules get enforced. Get any one of those wrong and the whole thing stalls, not gracefully, but all at once, usually at 2am when the target site's ops team notices your IP range and bans it mid-run.

The naive version everyone writes first looks like this: fetch a URL, parse the HTML, push any new links onto a queue, mark the URL as visited, repeat. It works fine at a few hundred pages. Somewhere between ten thousand and a million pages, four things break in roughly this order.

The visited set is the first casualty. A Python set holding every URL you've seen eats memory linearly, and there's no version of "just use a bigger machine" that survives contact with billions of URLs. Sequential fetching is next: one slow or unresponsive host parks your entire pipeline behind it, since nothing else can proceed until that request times out or returns. Then politeness catches up with you, because without rate limiting per domain, a single target bans your IP before the crawl even gets going. Last, there's no recovery story. Crash the process and every URL you'd queued, every bit of frontier state, is gone.

A production-grade distributed crawler operates at a completely different order of magnitude: tens of thousands of requests per second, sustained, with raw content storage measured in petabytes rather than gigabytes. That gap, between a script that runs on a laptop and a system that runs across a fleet, is the entire subject here. Going distributed doesn't just mean "add more workers." It means you now own five independent subsystems, frontier, fetcher, parser, deduplication, and politeness enforcement, each with its own failure modes and its own tradeoffs. The rest of this piece walks through each one.

The five-layer stack a distributed crawler is made of

Strip away the marketing language and a distributed crawler is a pipeline with five distinct layers, and each one needs to scale on its own terms.

Layer 1, data acquisition. Crawlers, headless browsers, or direct API calls pull raw content off target resources. Everyone pictures this when they hear "web crawler," but it's one piece of five.

Layer 2, network access. Requests get routed through IP pools, proxy infrastructure, or geographically distributed nodes. This layer exists almost entirely because of anti-bot defenses and rate limits.

Layer 3, parsing. Raw HTML, JSON responses, or a rendered DOM get turned into structured output, or into something clean enough to feed an LLM pipeline downstream.

Layer 4, validation. Deduplication, field normalization, and completeness checks happen here. This layer decides whether your dataset is trustworthy or just large.

Layer 5, storage and delivery. Structured data lands somewhere permanent, such as a database, a search index, a vector store, or object storage, before reaching whatever consumes it, including RAG systems and plain analytics jobs.

The principle that holds this together: each layer scales independently. If your parsers can't keep up, the fix is more parser workers, not more fetchers. Bolting on fetcher capacity when the parser is the bottleneck just moves the backup upstream, it doesn't remove it.

Sitting above all five layers is the frontier or queue manager, which actually runs the show. It holds the prioritized list of URLs, enforces politeness rules per domain, and hands out work to fetchers. In practice this gets built on Kafka, RabbitMQ, or a Redis-backed queue, and which one you pick shapes almost everything downstream of it.

URL frontier design: sharding, priority scoring, and tiered storage

A frontier is not a queue with a fancier name. A plain first-in-first-out queue can't do three things a real frontier has to do: hold billions of URLs without running out of memory, order work by priority instead of arrival time, and survive a crash without losing state.

Priority scoring usually weighs a few factors together: some measure of authority like PageRank, how long a URL has been sitting unfetched, and how likely the page is to have changed. PageRank (or some cheaper proxy for it) captures authority, so important pages jump the line. Age penalizes URLs that have been sitting in the frontier a long time without getting fetched, which stops the crawl from starving low-priority pages forever. Freshness rewards pages likely to have changed since the last visit. Weight these differently depending on what the crawl is actually for: a news crawler leans hard on freshness, an academic or reference crawl leans on authority instead.

At real scale, you can't keep the whole frontier in RAM. The fix is tiered storage: hot URLs, the ones about to be fetched, live in Redis. Warm and cold URLs spill to disk or object storage. Maintaining priority order across billions of entries without blowing your memory budget means using external sorting and merge strategies, the same techniques databases use to sort data sets larger than available RAM.

The fault-tolerance detail that matters most here: leases, not deletes. When a worker claims a URL, the frontier doesn't remove it, it leases it with a timeout. If the worker finishes successfully, it acknowledges the fetch and the URL comes off the frontier for good. If the worker crashes mid-fetch, the lease simply expires and the URL resurfaces for another worker to grab. No manual cleanup, no lost work.

Backing the frontier with Kafka buys you a durability guarantee almost for free: if the frontier service itself crashes, it recovers from the last committed offset instead of rebuilding its entire state from scratch.

A naive single-node crawler doing plain HTTP fetches runs somewhere around 60 to 500 pages per minute. At an average of 4 seconds per page for a single-node crawler, one worker simply cannot get anywhere near the throughput a production system needs. That gap is the entire justification for distributing the frontier across many stateless fetchers in the first place.

Consistent hashing and domain-based sharding: why politeness forces the partition key

Once you split the frontier across multiple shards, two different shards can independently decide to hit the same domain at the same time. The answer, if you don't design around it, is that you blow through that domain's rate limit and get banned, even though no single shard thought it was doing anything wrong.

Politeness is fundamentally a per-host property. The rate limit, the next-allowed-request time, the robots.txt rules, all of it needs to live in one place per domain. Sharding the frontier by a hash of the domain solves this cleanly: every URL for a given host lands on the same shard, so politeness gets enforced locally, with zero coordination needed between shards.

Consistent hashing is what makes this survive a growing or shrinking cluster. Add or remove a shard, and only a fraction of the keyspace needs to get remapped, not the whole thing. Without consistent hashing, resizing the cluster means reshuffling everything, which is expensive and slow at exactly the moment you're trying to add capacity.

Each shard owns its own robots.txt cache and its own per-domain token bucket, and there's no shared mutable state across shards to fight over. The token bucket itself is a simple mechanism: each host gets a bucket of tokens, every request drains one, and the bucket refills at whatever rate that host's rules allow. This handles both burst limits and sustained-throughput limits in one mechanism.

robots.txt is easy to get wrong in two opposite directions. Treat it as optional and you'll flood a site with disallowed requests and get flagged fast. Cache it forever and you'll miss paths a site newly opened up, or keep hammering paths it just closed off. The right approach is a per-domain cache with a real expiry.

Domain sharding can create hot shards. If a handful of high-value domains dominate your frontier, whichever shard owns those domains ends up doing disproportionate work. Sub-domain hashing, or a hard cap on how much priority any single domain can accumulate, keeps that imbalance in check.

Stateless fetcher workers and the limits of linear throughput scaling

Fetcher workers should hold no state at all. Pull a URL off the frontier, fetch it, hand the raw HTML to the parser queue, repeat. Because there's no local state to worry about, workers can be added, removed, or replaced without any coordination between them, and that's the whole point of the design.

Throughput scales roughly with worker count, at first. Add workers, get proportionally more pages per minute. That relationship doesn't hold forever, though. Push worker count high enough and the curve bends away from linear, because per-domain rate limits become the real ceiling long before raw fetch capacity does. Adding a thousandth worker doesn't help if a large chunk of your existing workers are already sitting idle, waiting on the same rate-limited domain.

Fixing that bottleneck means sharding the frontier by domain hash (already covered above), partitioning the deduplication filter the same way, and moving queue infrastructure over to something like Kafka that can actually absorb the coordination load.

Retries need real handling, not an afterthought. Transient failures, 5xx errors, timeouts, get exponential backoff. URLs that exhaust their retry budget go to a dead-letter queue instead of vanishing silently, because a failure you can't see is worse than one you can. And workers need to tell the difference between a slow host, which just needs backing off, and a blocked IP, which needs escalation to proxy rotation instead.

JavaScript-heavy sites add another wrinkle. Single-page apps and dynamically loaded content don't show up in a plain HTTP fetch, so headless browser automation, tools like Playwright or Puppeteer, has to step in for those targets. A headless browser worker eats far more CPU and memory than a plain HTTP fetcher doing the same job. The fix isn't running headless everywhere, it's classifying targets ahead of time and mixing headless and HTTP-only workers so you're not paying browser-rendering costs on static pages that never needed it.

Within each worker, async concurrency, asyncio-style patterns or their equivalent, lets one worker hold many requests in flight at once instead of blocking on each one sequentially. On high-latency targets, this cuts ingestion time meaningfully without needing to spin up more worker processes at all.

Deduplication at scale: Bloom filters, content hashing, and the false-positive tradeoff

Deduplication is actually two separate problems wearing one name. Have you already queued or fetched this exact URL? That's frontier-level dedup. Is this page's content identical, or close enough, to something already sitting in storage? That's parser-level dedup. Solving one doesn't solve the other, and treating them as the same problem is how duplicate content sneaks through.

For URL dedup, a Bloom filter does the job cheaply. It's a fast, probabilistic membership check: if it says "never seen this," you go do the expensive lookup against the real visited store. If it says "seen it," you skip the lookup entirely. The tradeoff is a small, tunable false-positive rate: it might occasionally tell you a genuinely new URL has already been seen, in exchange for a massive reduction in memory use compared to keeping a full set in Redis. The false-positive rate gets set at construction time, through the size of the bit array and the number of hash functions, so engineers choose their own acceptable skip rate rather than inheriting a fixed one. And the filter itself needs periodic snapshots to disk. Restart without one, and the only option is rebuilding the entire filter from zero, which throws away everything the crawl learned before the crash.

Content dedup works differently and lives at the parser layer, not the frontier. After normalizing a page's content, compute a hash of it, MD5 or SHA-256, and store that hash in a distributed wide-column store like Cassandra. This is what catches mirror pages, syndicated articles running on ten different domains, and soft-404s that return a success status but share content with some other error page entirely.

For distributed operation, partition the Bloom filter using the same domain hash that shards the frontier. Each shard owns its own slice of the filter, which means no shard ever needs to make a cross-shard call just to check whether a URL's been seen.

A false-positive rate that occasionally skips a legitimate page is fine for broad, exploratory crawls, but it is not fine for compliance monitoring or change-detection pipelines. A false-positive rate that occasionally skips a legitimate page is fine for broad, exploratory crawls. It is not fine for compliance monitoring or change-detection pipelines, where missing a real update is the failure mode you're specifically trying to avoid. In those cases, the full visited-set lookup can't be replaced by a probabilistic shortcut, and that's a decision to make on purpose, not a corner to cut by accident.

Parser scaling, backpressure, and keeping the pipeline from inverting

Fetchers can deliver pages faster than parsers can process them, causing the queue between the two layers to grow without bound, and eventually fetchers stall waiting for room in a queue that never drains, a failure mode that resembles a fetcher problem without being one. Throughput collapses even while every fetcher worker is technically healthy and running. Adding more fetchers at that point makes the backlog worse, not better.

Backpressure is the actual fix, not more fetcher capacity. Put Kafka or a Pub/Sub layer between fetchers and parsers, since that gives you durable buffering and the ability to replay messages if something downstream fails. When the parser consumer group starts falling behind, the system needs to slow fetcher admission before the queue saturates, not after the damage is already visible.

Parser workers scale the same way fetchers do: stateless, horizontally, independent of how many fetchers exist. If parsing is the slow part, add parser replicas. Fetcher count doesn't need to move at all.

Async processing decouples the two sides cleanly. A fetcher acknowledges delivery to the queue and immediately moves to its next URL, without waiting around for a parser to actually process anything. The parser consumes at whatever rate it can sustain, on its own schedule.

Parsers feed back into the loop that keeps a crawl expanding: newly discovered URLs, with link metadata attached for priority scoring, go back to the frontier, while structured content or raw HTML gets forwarded to storage. One monitoring signal earns a permanent spot on a dashboard: queue depth between fetcher output and parser input. A rising trend in queue depth between fetcher output and parser input appears before throughput actually starts visibly degrading, making it the earliest warning sign of parser under-provisioning.

Proxy infrastructure and anti-bot evasion: where architecture meets operational cost

Anti-bot detection has moved well past checking user-agent strings, and a lot of teams still design around the old assumption. Target sites now flag datacenter IP ranges directly, fingerprint headless browsers by their rendering quirks, and watch behavioral signals like mouse movement or request timing that no amount of header spoofing fixes. Research has documented how datacenter proxies and headless-browser artifacts get caught by exactly these signals, independent of whatever headers a request carries.

Once a site blocks an IP range, there are really only two paths forward: pay for residential proxy pools, which cost meaningfully more, or accept that your success rate against that target is going to keep dropping.

Which proxy strategy fits depends entirely on the target. Datacenter proxies are fine for permissive sites where raw cost and throughput take priority over staying invisible. Rotating residential proxies, where requests appear to come from real residential IP addresses, become necessary once a target has serious detection in place. Geo-distributed nodes matter when a target serves different content by region, or when latency to a regional target actually affects crawl speed.

Building and maintaining a custom proxy rotation layer in-house becomes an ongoing operational cost if a pipeline needs a high volume of requests per hour against targets that are actively hardened against bots, raising a build-versus-buy question. If a pipeline needs a high volume of requests per hour against targets that are actively hardened against bots, building and maintaining a custom proxy rotation layer in-house becomes an ongoing operational cost, one that often runs higher than a managed subscription would. This is the same argument that applies to most of the crawling stack: infrastructure that doesn't differentiate your product is rarely worth owning yourself.

Proxy selection also needs to stay domain-aware, tied to the same shard assignment used for the frontier. Requests to a single domain need to come from a consistent proxy pool within a session, since a domain seeing requests jump between wildly different IP ranges is itself a signal that gets sites flagging the traffic.

Fault tolerance patterns that keep a distributed crawl running through node failures

The whole architecture rests on one design principle: each node handles capture, parsing, and processing end to end, so one node failing doesn't stop the crawl. It just reduces throughput until that node either recovers or gets replaced. Nothing has to halt and restart from scratch because one machine died.

Kafka's committed-offset model is what makes frontier durability real rather than theoretical. If the frontier service crashes, it picks back up from the last committed offset instead of losing whatever URLs were queued at the time. That durability guarantee is the specific reason Kafka gets chosen over an in-memory queue for backing the frontier, not just a general preference for one messaging system over another.

Lease semantics do the same job for fetcher crashes that Kafka does for frontier crashes. A worker that dies mid-fetch simply lets its lease expire, and the URL it was holding resurfaces automatically for another worker to pick up. No manual intervention, no lost URL, no separate recovery process to run.

And the Bloom filter snapshots to disk address the last remaining gap. Without them, a restart means rebuilding the entire visited-set filter from nothing, which throws away everything the crawl already learned. With snapshots in place, recovery picks up close to where the crash happened instead of starting over.

Put these four patterns together, leases instead of deletes, offsets backed by a message queue, domain-sharded politeness, and Bloom filter snapshots, and node failures stop being crawl-ending events. They become throughput dips. That distinction is the entire difference between a script that happens to run on several machines and a system built to survive running on them.