A Crawl Is a Long-Running Process

Scrapes fail. The target goes down, the container restarts, the network blips, the disk fills, the IP gets throttled. If every failure means restarting from page one, you have built a job that punishes you on schedule. The fix is to make the crawl idempotent: cache what you have already fetched, dedupe your work queue, and persist enough state that the next run picks up exactly where the last one died.

Cache the Raw HTML

The first and cheapest layer is a raw-HTML cache. Parse against a local copy during development, and at runtime, never hit the network twice for the same URL. A file-per-URL layout is simple and debuggable: the cache key is a hash of the URL, the value is the body, and the file system is the index.

import hashlib
import pathlib

CACHE = pathlib.Path("html_cache")
CACHE.mkdir(exist_ok=True)

def cache_key(url: str) -> str:
    return hashlib.sha256(url.encode("utf-8")).hexdigest()

def get_page(client, url: str) -> str:
    path = CACHE / (cache_key(url) + ".html")
    if path.exists():
        return path.read_text(encoding="utf-8")
    resp = client.get(url, timeout=15.0)
    path.write_text(resp.text, encoding="utf-8")
    return resp.text

Cache every page, even ones with errors, under a separate suffix or with the status code in the name. A page that 404ed is data too: without it you will retry the URL forever.

Dedupe the Work Queue

The cache stops network waste. A separate dedupe stops processing waste. Keep a set of every URL you have started, and filter it before you add new links to the queue. Normalize URLs first or the same page sneaks in under five spellings: strip the fragment, drop utm_* query parameters, sort the remaining query parameters, and lowercase the host.

from urllib.parse import urlsplit, urlunsplit, parse_qsl, urlencode

def normalize(url: str) -> str:
    parts = urlsplit(url)
    query = sorted((k, v) for k, v in parse_qsl(parts.query)
                   if not k.startswith("utm_"))
    return urlunsplit((parts.scheme, parts.hostname,
                       parts.path, urlencode(query), ""))

A seen table backed by SQLite with a UNIQUE constraint on the normalized URL makes the dedupe survive restarts for free, because you can query it out of the box instead of re-discovering the crawl's memory.

Persist the Queue

In-memory queues vanish on a crash. For any crawl bigger than a few thousand pages, persist the queue in a table with states: pending, in_progress, done, failed. Each worker claims a chunk of pending rows, marks them in_progress, completes them, and updates to done. On restart, simply query all pending and failed rows and continue. The database write doubles as a progress report.

import sqlite3

def claim_batch(con, size=50):
    rows = con.execute(
        "SELECT url FROM queue WHERE state='pending' LIMIT ?", (size,)
    ).fetchall()
    for (url,) in rows:
        con.execute("UPDATE queue SET state='in_progress' WHERE url=?", (url,))
    con.commit()
    return [url for (url,) in rows]

Journal Extracted Records

When a crash loses your in-memory result list, it also loses the work that already succeeded. Append each parsed record to a JSONL file the moment it is produced, and keep a written marker in the same row as the URL. Recovery then only redoes the pages you never finished.

import json

def save_record(con, url, record):
    with open("out.jsonl", "a", encoding="utf-8") as f:
        f.write(json.dumps(record) + "\n")
    con.execute("UPDATE queue SET state='done' WHERE url=?", (url,))
    con.commit()

Design for Restart From Day One

Write the resume path before the crawl, not after the second crash. The three rules: fetch through the cache, claim work from a persistent queue, and journal results as you go. Once those are in place, a "crawl" is just a loop that keeps claiming batches until the queue is empty, and a "crash" is just a pause.