One Process Stops, You Lose It All

Everything so far has been a one-process crawl. That is the right size for many jobs, but the moment your backlog grows into the hundreds of thousands, three problems appear at once. A single process crashes and your supercomputer-grade cache lesson still has to re-run everything it was in the middle of. A single process is bound to one machine, so its concurrency is capped by that machine's sockets and its rate by one IP. And a single process is one place to fail, which is one thing you cannot afford once the crawl has been running for days.

Spreading the crawl across workers means separating the two things that are currently tangled together: the state (what is left to do, what is done, what is being worked on right now) and the execution (the fetching and parsing). Move the state into a shared store that any worker can see, and workers become interchangeable, stateless, crash-recoverable, and scalable to as many machines as you can rent.

The Reliable Queue

The backbone is a queue with an honest notion of "who is working on this item." Redis is the workhorse here because the operations are atomic and are exactly the ones you need. The reliable pattern is brpoplpush: pop an item onto a processing list with a lease timeout, and remove it only when finished. If a worker dies mid-item, the item sits in processing until its lease expires and a recovery pass returns it to the todo list.

import redis

r = redis.Redis(decode_responses=True)

def claim(lease=120):
    item = r.brpoplpush("crawl:todo", "crawl:processing", timeout=5)
    if item:
        r.expire("crawl:processing", lease)
    return item

def finish(item):
    r.lrem("crawl:processing", 1, item)
    r.sadd("crawl:done", item)

def recover():
    for item in r.lrange("crawl:processing", 0, -1):
        r.lrem("crawl:processing", 1, item)
        r.lpush("crawl:todo", item)

The r.sadd in finish gives you a third list, crawl:done, as the distributed version of the dedupe set from the caching lesson. Workers check done before enqueuing links, and the queue stays rejection-free even when 40 workers push at once.

Divvying State: The Lease

The lease is the contract. Two workers must never fetch the same URL for the same reason; the creator of the queue must decide whether duplicates are tolerable. In the brpoplpush pattern, only one worker ever holds an item because the pop and push happen atomically, and the lease only decides what happens on a crash. Pick a lease slightly longer than the slowest conceivable item. When a worker finishes an item, call finish before the next claim, so a worker that dies between the two steps leaves a clean retirement.

Coordination Versus Rate: The Shared Budget

Your workers now share the target's patience with no shared memory. The per-worker token bucket from the concurrency lesson silently multiplies with each worker, and a fleet of 12 silent buckets is a 12x overshoot nobody authorized. Centralize the budget in the shared store with a simple sliding window: one key per target per minute, incremented atomically.

import time

def allow(client_r, key, max_per_minute):
    window = int(time.time()) // 60
    rkey = "rl:" + key + ":" + str(window)
    client_r.setnx(rkey, 0)
    count = client_r.incr(rkey)
    if count > max_per_minute:
        time.sleep(2)
        return False
    return True

The minute window is coarse but it is honest, cheap, and does not require a Lua script. When the window resets the crawl resumes. If you need smoother pacing, the token bucket becomes a Lua script or a lease on per-second counters, but start with the window and upgrade only when the numbers actually hurt.

Graceful Shutdown

Workers die by signal or be killed. A shutdown handler is the difference between a queue that recovers in seconds and one that re-fetches tens of thousands of in-flight URLs after every deploy. Catch SIGTERM, stop claiming, finish the item you are on, and let the runtime exit.

import signal

stopping = False

def _stop(signum, frame):
    stopping = True

signal.signal(signal.SIGTERM, _stop)

Every worker that exits cleanly instead of mid-request leaves the queue in a state the next cloud of replacements can pick up instantly. Pair that with a heartbeat row (worker id, last-seen timestamp) and a reducer that flags dead workers whose leases are about to expire, and your "crawl" begins to look like a small, self-healing system instead of a fragile script. That is the whole point of the distributed step: the crawl stops being a thing you hope completes and becomes a thing you can watch, restart, and observe.