Throughput without Chaos

If one request takes 0.3 seconds, a sequential crawl of 100,000 pages takes nine hours no matter how fast your parsing is. Most of that time is waiting on the network, and waiting is where a single thread is wasted. Parallelism lets the CPU sit idle less and the pipe stay full. The catch is that unbounded parallelism gets you blocked, banned, and rate-limited faster than anything else, so the skill is coupling concurrency with the controls from the last lesson.

Why asyncio Fits Scraping

Scraping is I/O-bound. You send a request, the server thinks, the bytes travel, and meanwhile your program is doing nothing. asyncio turns that nothing into an opportunity: while one request waits, another one runs. One OS thread can hold hundreds of in-flight requests, which is far lighter than the same number of threads. httpx ships a first-class async client, so you do not need a separate library.

A Contained Worker Pool

The building block is a semaphore that caps how many requests can be in flight at once. Everything else stays simple: gather futures, limit concurrency, collect results.

import asyncio
import httpx

async def fetch(client, url, sem, out):
    async with sem:
        try:
            resp = await client.get(url, timeout=15.0)
            out.append((url, resp.status_code, resp.text))
        except httpx.HTTPError as exc:
            out.append((url, "error", repr(exc)))

async def crawl(urls, workers=10):
    sem = asyncio.Semaphore(workers)
    out = []
    limits = httpx.Limits(max_connections=workers,
                          max_keepalive_connections=workers)
    async with httpx.AsyncClient(limits=limits, timeout=15.0) as client:
        await asyncio.gather(*(fetch(client, u, sem, out) for u in urls))
    return out

results = asyncio.run(crawl(all_urls, workers=10))

The semaphore is the ceiling. Set it from data, not from intuition: what rate did the site tolerate yesterday? A TokenBucket from the previous lesson makes the ceiling explicit (requests per second) and the semaphore simply keeps that many in flight. Typical numbers are 5 to 20 concurrent requests, not hundreds, for most public sites.

Connection-Level Control

httpx.Limits decides how many TCP connections and keep-alive sockets the client holds. Keep max_connections in the same ballpark as your workers. If you try to push 500 workers through 10 connections, requests pile up behind the connection pool and your concurrency is fake. If you leave max_connections at the default of 100 while scraping with a careful token bucket, you have handed yourself 100 open sockets to a server that wanted two per second. Bound both.

Error Isolation

A single dead host can sink a whole gather if an exception escapes the coroutine. Catch per-request exceptions inside the worker, record the failure, and re-queue the URL later. Treat 429 as a global signal, not a per-request one: when any worker sees one, lower the concurrency ceiling for the whole crawl, pause, and resume when the Retry-After window passes.

async def retry_policy():
    await asyncio.sleep(60)   # global cooldown after repeated 429s

When Parallelism Backfires

Parallelism only helps when the bottleneck is your side of the pipe. It makes things worse when the bottleneck is the target: a per-IP rate limit, a login rate limit, or a shared backend. If the site is fast and the API is pleasant, one careful worker with a token bucket is often all you need, and it is the cheapest and least likely to be blocked. Add concurrency only after measuring that you are request-starved, and always bring a token bucket and a Retry-After handler along for the ride.

Memory Discipline

Holding every response body in a list is fine for a thousand pages and a disaster for a million. Prefer writing to disk or a database inside the worker, and stream pages you only need to inspect once with client.stream(). Process rows as they arrive instead of collecting the whole world into RAM, and the crawl stops having a size limit.