Stop Re-Scraping Everything

The naive crawler re-downloads the entire site on every run. Fine at two hundred pages, insulting at two million, and it guarantees your crawl spends the whole day fetching pages whose content did not change. The mature design flips the question: fetch only the pages that changed, and let the unchanged ones skip the loop entirely. This lesson wires that habit into the storing and family of lessons with three cooperating signals: server hints, content hashes, and structured diffs.

Signal One: HTTP Conditional Requests

HTTP ships with change detection built in. A response carries either an ETag (an opaque version string) or a Last-Modified date. On your next request, send the value back in If-None-Match or If-Modified-Since. If nothing changed, the server answers 304 Not Modified with no body, and you have saved the entire transfer.

import httpx

client = httpx.Client(timeout=15.0)
etag_by_url = {}

def fetch_if_changed(url):
    headers = {}
    etag = etag_by_url.get(url)
    if etag:
        headers["If-None-Match"] = etag
    resp = client.get(url, headers=headers)
    if resp.status_code == 304:
        return None  # unchanged, reuse cached copy
    etag_by_url[url] = resp.headers.get("ETag", "")
    return resp.text

ETag beats Last-Modified on nearly every site that offers both, because it is exact: a server that generates a fresh Last-Modified for every request defeats date comparisons, and some CDNs bump the timestamp when nothing else changes. When a site supports neither header, you fall back to signal two.

Signal Two: Content Hashing

Without server hints, the crawler computes the fingerprint itself. Hash each response and skip storage when the hash matches the previous run. The substitution cost is a whole page re-transfer instead of just headers, but the save compared to full re-crawls is still orders of magnitude.

import hashlib

hash_by_url = {}

def changed_response(url, body):
    digest = hashlib.sha256(body).hexdigest()
    if digest == hash_by_url.get(url):
        return False
    hash_by_url[url] = digest
    return True

Run the hash over the normalised bytes before parsing so that insignificant byte jitter (a session id in the markup) does not count as change. If the noise is too high, move the comparison one layer deeper.

Signal Three: Structured Diff

Hashing whole pages cannot see that "3 in stock" became "0 in stock" if the surrounding page also gained a timestamp byte. When your pipeline cares about field-level change, compare the parsed, normalised record (from data-cleaning-validation) against the stored one. The canonical statement of "did this item change" is then a SQL upsert: the ON CONFLICT path either updates the row or leaves it untouched, and you can count which happened.

INSERT INTO items (sku, title, price, updated_at)
VALUES (?, ?, ?, ?)
ON CONFLICT(sku) DO UPDATE SET
    price = excluded.price,
    updated_at = excluded.updated_at,
    changed = 1
RETURNING sku;

Sitemaps and lastmod as the Cheap Filter

The sitemap lesson gave you the site's own claims about what changed. The incremental crawler folds that in cheaply: skip any URL whose sitemap lastmod is older than your last run, and let conditional requests be the safety net for the sitemap's lies. Order of operations for a fresh run: sitemap lastmod filter first, then conditional headers, then hashing, then structured diff at the edge. Each layer catches what the one above missed, and the pipeline pays only for the layers it actually needs.

The Modernist Alternative: Listen Instead of Poll

For fast-moving sources, polling is a status-quo you maintain. When a site runs its own feed (from the feeds lesson) or WebSocket stream (from the real-time lesson), consuming the push is natively incremental: you store what arrives and never poll anything. Incremental scraping is not a single trick, it is the habit of always asking "what is the smallest set of bytes that tells me something changed" before writing a bulk re-fetch.