Throttle Like the Site Owns the Server

The internet is a shared pile of other people's infrastructure. A scraper that sends one request per millisecond might succeed for an hour, then meet a WAF, a bruised server, or a blocked subnet. The code that lasts is the code that acts like a conscientious user: it paces itself, respects explicit signals, and returns to a sensible pace when the server complains. Politeness is not separate from results. Politeness is what keeps the pipeline alive long enough to get results.

The Token Bucket

The cleanest throttle is a token bucket. The bucket fills at a steady rate and holds a small burst, and each request spends a token. This gives you a smooth average rate with room for short bursts, which is exactly how a human behaves.

import asyncio
import time

class TokenBucket:
    def __init__(self, rate: float, burst: int):
        self.rate = rate          # tokens per second
        self.burst = burst
        self.tokens = float(burst)
        self.updated = time.monotonic()

    async def acquire(self):
        while self.tokens < 1.0:
            now = time.monotonic()
            elapsed = now - self.updated
            self.updated = now
            self.tokens = min(self.burst, self.tokens + elapsed * self.rate)
            if self.tokens < 1.0:
                await asyncio.sleep(0.05)
        self.tokens -= 1.0

bucket = TokenBucket(rate=2.0, burst=5)

async def fetch(client, url):
    await bucket.acquire()
    return await client.get(url)

A rate of two requests per second is a defensible baseline for most public sites. Read the target's robots.txt first; a Crawl-delay directive there is the operator telling you their tolerance.

import urllib.robotparser

rp = urllib.robotparser.RobotFileParser()
rp.set_url("https://example.com/robots.txt")
rp.read()
delay = rp.crawl_delay("*")
if delay is not None:
    print("server asks for", delay, "seconds between requests")

Honor Retry-After

When a server returns 429 Too Many Requests, it usually tells you exactly how long to wait in the Retry-After header. Believe it. Sleeping for the stated delay and trying again resolves a huge fraction of rate-limit problems without any proxy spend.

import time

def polite_get(client, url, retries=4):
    for attempt in range(retries):
        resp = client.get(url, timeout=15.0)
        if resp.status_code == 429:
            header = resp.headers.get("Retry-After", "60")
            delay = int(header) if header.isdigit() else 60
            time.sleep(delay * (1 + 0.2 * attempt))  # backoff grows per retry
            continue
        return resp
    resp.raise_for_status()
    return resp

For 5xx responses, the server is overloaded or failing, not necessarily angry at you. Back off exponentially, add jitter so a fleet of your own workers does not stampede in lockstep, and give up after a handful of attempts instead of hammering a dying service.

Put the Throttle at the Right Layer

Keep the throttle close to the network calls, not around your parsing or your database writes. If the throttle sits too high, you throttle work that never touches the network and you leave the actual requests unsynchronized. In an asyncio crawl, share one bucket across all workers, because each worker holding its own bucket multiplies the true rate by the number of workers.

When the Site Says Stop, Stop

Some operators block you permanently and some respond with a block page that looks like a legal page. A polite scraper reads the response body on suspicious status codes, because a 200 with "We noticed unusual traffic" is a block in disguise. Detect those signatures, pause the whole crawl, back off for a long window, and only resume when the target is reachable again. A crawl that pauses for an hour and then resumes has cost you an hour. A crawl that keeps hammering a blocked target has cost you the target.