Put the Whole Course on One Target

Every lesson so far taught a technique in isolation. This capstone forces them into one system. The assignment: monitor a product catalog, keep a clean database, fetch only what changes, never get blocked, and survive a crash silently. That single sentence is the entire course. Here is the assembly order, with the lesson each step leans on spelled out, so you finish with a mental map of the whole pipeline.

Step 1: Reconnaissance (sitemaps-and-feeds, mitmproxy-and-har, robots)

Open robots.txt. Read its Sitemap: lines and download the sitemap. Check for an RSS feed. If the catalog is API-backed, capture a session and replay the JSON endpoints (mitmproxy or HAR). You now own the site's inventory without a single selector.

Step 2: Polite Fetching (rate-limiting, caching, error-handling)

Lock a session, set the TokenBucket to a reasonable rate, apply the retry-with-backoff wrapper, and mark every URL processed. Persist the queue with the crash-resume machinery: cache raw HTML, journal every record to disk as it is produced. This is the layer that keeps a three-week crawl alive.

Step 3: Extraction and Normalization (selectors, xpath, parsing, cleaning)

Parse with lxml or BeautifulSoup+lxml, choose CSS for visual targets and XPath for anything relational, resolve the encoding before parsing (mojibake is the silent killer), and normalize every field that will be compared later: price, date, id, status. Add the completeness check that flags a layout drift before the database fills with Nones.

Step 4: Scope the Fetch to What Changed (incremental-scraping)

Use sitemap lastmod as a cheap prefilter, conditional headers (If-None-Match) as the exact check, and content-hash diffing as the fallback for sites with no hints. The crawl shrinks from "everything every time" to "the delta each time".

Step 5: Store and Dedupe (storing-data, incremental)

Land the records in SQLite with a schema that is honest about keys: one row per SKU (or canonical URL), an upsert that updates price and touches updated_at only when a value actually changed, and a freshness query you can run by hand. The MAX(updated_at) per source count from the monitoring lesson is the health status of your whole system.

Step 6: Raise the Ceiling Only When Measured (concurrency, proxies)

Start sequential and prove a bottleneck before introducing asyncio or rotating proxies. Add a semaphore-guarded pool with a shared token bucket, add proxies only if an IP actually gets throttled, and keep sessions sticky. Every speed increase must be traceable to a measurement; otherwise you have added a failure mode without a benefit.

Step 7: Automate and Watch (scheduling-and-monitoring, scrapy)

Put the entry point on cron with a lock file, write a status row each run, and wire staleness / volume-collapse / error-spike alerts to a webhook. If the target is a long-lived property, move the pipeline to Scrapy and let its scheduler, dupe filter, auto-throttle, and job persistence replace the hand-rolled equivalents.

The Rubric That Says "Done"

A scraper is finished when it satisfies five checks: it obeys robots.txt and the ethics/legal floor from those lessons; it re-fetches only changed content and resumes after a crash; it never hammered a host past the rate the host tolerated; its database passes the freshness and completeness queries; and a dropped run produces an alert, not a mystery. Build against that list and you have production scraping. Build it against the techniques in isolation and you have a script that works until the first weekend.