The Part That Isn't HTML

Data does not end at HTML. Companies publish CSVs of open prices, archives of annual reports, full-disk images, and gigabytes of logs. Downloading these efficiently is a different skill from parsing pages: it is about memory, resumability, and verification, because the failure modes are "out of memory", "network dropped at 87%", and "the file is corrupt and nobody noticed." Three patterns cover virtually every binary download a scraper faces.

Stream in Chunks, Never In Full

resp.text and resp.json() load the whole payload into memory, which is fine for a page and a slow-motion catastrophe for a 2 GB archive. With stream=True, httpx hands you the body progressively and you write it to disk in bounded chunks. Memory stays flat no matter how large the file.

import httpx

def download(client, url, dest):
    with client.stream("GET", url) as resp:
        resp.raise_for_status()
        with open(dest, "wb") as f:
            for chunk in resp.iter_bytes(chunk_size=65536):
                f.write(chunk)

The 8192-sized default is a good start, but tune the chunk to the token: local disks like bigger chunks, and a 64 KiB default balances throughput against RAM. Never accumulate chunks into a list and join them at the end; that is just slower resp.content.

Verify as You Write

Corruption arrives silently, and a corrupt corpus costs you the same as a missing one. Hash the stream while writing, then compare against the digest the site published (downloads pages usually print one). Two birds: you never re-read the file, and the hash check runs in the same loop.

import hashlib

def download_verified(client, url, dest, expected_sha256):
    h = hashlib.sha256()
    with client.stream("GET", url) as resp:
        resp.raise_for_status()
        with open(dest, "wb") as f:
            for chunk in resp.iter_bytes(chunk_size=65536):
                f.write(chunk)
                h.update(chunk)
    if h.hexdigest() != expected_sha256:
        raise RuntimeError("checksum mismatch for " + url)

Resume Interrupted Transfers With Range

Network drops are a law of nature. HTTP's answer is Range: the server, if it advertises Accept-Ranges: bytes, will start a response at a byte offset you name. A resumable download therefore tracks how many bytes already landed and restarts with a Range: bytes=N- header, appending to what is already on disk.

import os

dest = "big.zip"
done = os.path.getsize(dest) if os.path.exists(dest) else 0

headers = {"Range": f"bytes={done}-"}
with client.stream("GET", url, headers=headers) as resp:
    if resp.status_code == 206:  # partial content honored
        mode = "ab"
    else:
        mode = "wb"  # server ignored the range; start over
        done = 0
    h = hashlib.sha256()
    with open(dest, mode) as f:
        for chunk in resp.iter_bytes(chunk_size=65536):
            f.write(chunk)
            h.update(chunk)

Combine the hash from the previous snippet with resume and you get a downloader that survives restarts wholesale: it resumes over a broken connection, then re-verifies the finished file against the published checksum, raising only if the bytes really are wrong.

Parallel and Scheduled Downloads

For many independent files, fan out with the concurrency lesson: an asyncio pool, each coroutine stream()-ing to its own file handle, with the token bucket guarding the shared rate. For routine bulk pulls (daily price snapshots), wrap the same function in the scheduling-and-monitoring pattern. The discipline that transfers from page scraping is identical: bounded concurrency, polite pacing, resumable state, and a checksum you can swear by. The only real difference between a page and a payload is the size.