The Structured Doors Into a Site

Before writing a single selector, check whether the site publishes a machine-readable map of itself. Most do. Two formats dominate: XML sitemaps (a list of URLs with metadata) and RSS/Atom feeds (a stream of the newest content). Both turn "crawl the whole site" into "read the index", and both are dramatically kinder to the server than a blind crawl. The polite engineering move is to use them first and crawl second.

From robots.txt to the Sitemap

robots.txt — already covered in its lesson — usually points at the sitemap with a Sitemap: directive. Read it, fetch whatever it names, and you have the site's complete URL inventory.

import httpx
from xml.etree import ElementTree as ET

robots = httpx.get("https://example.com/robots.txt", timeout=15.0).text
sitemap_url = next(
    (line.split(": ", 1)[1] for line in robots.splitlines()
     if line.lower().startswith("sitemap:")),
    "https://example.com/sitemap.xml",
)

resp = httpx.get(sitemap_url, timeout=15.0)
root = ET.fromstring(resp.content)
ns = {"s": "http://www.sitemaps.org/schemas/sitemap/0.9"}

if root.tag.startswith("{http://www.sitemaps.org/schemas/sitemap/0.9}sitemapindex"):
    for sitemap in root.iter("{http://www.sitemaps.org/schemas/sitemap/0.9}sitemap"):
        child = sitemap.findtext("s:loc", namespaces=ns)
        print("nested sitemap:", child)
else:
    urls = [u.findtext("s:loc", namespaces=ns) for u in root.iter("s:url")]
    lastmod_dates = [u.findtext("s:lastmod", namespaces=ns) for u in root.iter("s:url")]

One wrinkle: sitemaps can be split into an index sitemap that points at dozens of child sitemaps (per section, per archive year). Detect the root element's tag and follow the children recursively rather than assuming a flat list.

The Sitemap Is a Contract

A lastmod value in a sitemap is the site telling you when a URL last changed. That single value powers the incremental-scraping lesson: fetch only URLs whose lastmod is newer than your last run, and skip the rest. Large sites publish sitemaps precisely so crawlers can be selective. Treat a sitemap as the authoritative inventory and your own crawl as the reconciliation layer that catches what the sitemap misses.

Feeds: the Change Stream

RSS and Atom exist because publishing should be a stream, not a scan. If the site offers a feed, you skip the whole "detect new items" problem: the feed lists the newest entries with titles, links, and timestamps, usually the last 10 to 50 items.

import feedparser

feed = feedparser.parse("https://example.com/feed.xml")
for entry in feed.entries:
    print(entry.title, entry.link, entry.get("published", ""))

Feeds are the single best entry point for news sites, blogs, and release notes. They are also honest about their failure modes: an RSS feed may trim teasers instead of full text, and some feeds miss items that appeared in the HTML crawl. Run the feed as the primary new-item discoverer and the sitemap as the completeness check.

Where to Find Them

Discovery order for any new target: the Sitemap: line in robots.txt; the <link rel="alternate" type="application/rss+xml"> and ...atom.xml links in the HTML head; then guess the conventional paths. Only after all three fail do you start from the homepage with a frontier crawl. Sites that hide their sitemaps are rare, because sitemaps help their own SEO; a missing sitemap is usually a signal the site moves fast and breaks quickly, which is a warning about its stability more than its propriety.