Crawling Is Prioritization

Scraping a single page is easy. Crawling a site means deciding, at every moment, which page to fetch next out of thousands of discovered-but-unvisited URLs. That decision is the URL frontier, and the choices you make there set your politeness, your coverage, and your time-to-first-value. Most scraping fails on strategy long before it fails on code: skipping the homepage to sprint into archives, hammering one host while starving another, or re-fetching the same list page fifty times.

The Frontier: Queue + Visited Set + Depth

Every crawler holds a queue of pending URLs, a visited set (the dedupe tool from the caching lesson), and a depth field for each discovered URL. The loop is universal: pop a URL, fetch it, parse the links, normalize and filter them, and push the survivors back with depth+1. What separates crawlers is the selection policy in that loop.

import httpx
from urllib.parse import urljoin

queue = [("https://example.com/", 0)]
visited = set()
client = httpx.Client(timeout=15.0)

while queue:
    url, depth = queue.pop(0)
    if url in visited:
        continue
    visited.add(url)
    if depth >= 4:
        continue

    resp = client.get(url)
    hrefs = resp.text  # in practice: parse, extract a[href], normalize
    for link in extract_links(resp.text):
        absolute = urljoin(url, link)
        normalized = normalize_url(absolute)
        if normalized and normalized not in visited:
            queue.append((normalized, depth + 1))

The visited check at the top is your correctness guarantee. The depth cap is your politeness and your budget: sites are infinite if you let them be.

Breadth First Is the Default

Depth-first wanders deep into one branch while ignoring that the site also has a trending-products section a homepage click away. Breadth-first keeps every discovered branch at roughly equal freshness, which is what humans expect and what list-and-detail crawls want. Traverse by appending to the queue (as above) rather than pushing onto a stack, and you get BFS for free. The exception is when the site you want lives entirely behind a "next" chain of detail pages; then the two-phase crawl below beats BFS.

The Two-Phase Crawl

The classic large-crawler shape is: phase one walks lists and indexes, collecting coupon-like detail URLs into a plain store without touching them; phase two fetches the detail pages with the concurrency machinery from the dedicated lesson, fully parallel. Doing it this way, rather than interleaving list and detail fetches, means phase-one politeness is gentle (a few hundred requests) and phase two runs at maximum throughput against a known list. You can even persist phase one's output with the resume machinery from caching-and-resume and separate the two phases in time entirely.

Per-Host Politeness

A single global queue lets one fast host hog the frontier while another sits forever. The professional design buckets the frontier by host, with each host getting its own delay budget, its own proxy, and its own failure state — drawn straight from the rate-limiting and caching lessons. Politeness decisions live at the host level, not the crawl level:

from collections import defaultdict

host_queue = defaultdict(list)
host_next_ok = defaultdict(float)  # hostname -> earliest allowed start time

def enqueue(url, depth):
    host = httpx.URL(url).host
    host_queue[host].append((url, depth))

def next_url():
    host, delay = min(host_next_ok.items(), key=lambda kv: kv[1])
    if time.time() < delay or not host_queue[host]:
        return None
    return host, host_queue[host].pop(0)

Selection Policy Is Your Product

The frontier can also encode priorities: recency (new items before old), relevance (category popularity), or freshness needs (prices before backlinks). Whatever the priority, encode it into the queue's pop rule once, at one place. Ever-changing ad-hoc partial ordering in the main loop is how crawlers get bugs that re-fetch half a site. Decide the policy, codify it, and let the loop stay boring.