Trade the Glue for a Framework

By this point you have hand-built every piece: sessions, retries, dedupe, throttling, caching, storage. That is the right way to learn, and a laborious way to operate. For a persistent pipeline, switch to Scrapy, the framework that commercial scrapers run on. Scrapy codifies the lessons you learned exactly where the reuse pays: it brings its own session management, robotstxt checker, retries, a strict URL dedupe filter, request throttling (AUTOTHROTTLE), an items pipeline, stat collection, and a pluggable middleware stack. Your job shrinks to writing the site-specific parts.

The Shape of a Project

scraper/
    spiders/
        products.py
    items.py
    middlewares.py
    pipelines.py
    settings.py

The web of it: a Spider yields either Requests (work) or Items (results); items.py declares the record's fields; pipelines.py cleans, dedupes, and stores items in order; middlewares.py wraps requests (headers, proxies, parsing engines); settings.py governs concurrency and throttling.

A Minimal Spider

import scrapy

class ProductsSpider(scrapy.Spider):
    name = "products"
    start_urls = ["https://example.com/products"]

    def parse(self, response):
        for card in response.css(".product"):
            yield {
                "title": card.css("h3::text").get(),
                "price": card.css(".price::text").get().strip(),
            }
        for href in response.css(".pagination a::attr(href)"):
            yield response.follow(href, callback=self.parse)

Everything you built manually is now default: response drops retries and redirects in your lap, follow handles relative URLs and the referer, the DUPEFILTER stops the same href twice, and each yield of a Request feeds the scheduler — which is the frontier lesson, provided by the engine.

Pipelines Make Clean Record Hitting the Database

Pipelines run in the order of their numeric key. The canonical stack is, in order: validate, clean/normalise (the data-cleaning lesson), dedupe, then store.

class PriceCleaner:
    def process_item(self, item, spider):
        item["price"] = float(re.sub(r"[^0-9.]", "", item["price"] or ""))
        return item

And in settings.py, register it, then let item exporters serialize either via FEEDS or a storage pipeline.

ITEM_PIPELINES = {"scraper.pipelines.PriceCleaner": 100}
DOWNLOAD_DELAY = 0.5
AUTOTHROTTLE_ENABLED = True
AUTOTHROTTLE_MAX_DELAY = 5.0
CONCURRENT_REQUESTS_PER_DOMAIN = 8

AUTOTHROTTLE is the rate-limiting lesson, automated: it watches your own latency and slows down when the server starts hurting — the polite-crawling lesson, without the plumbing.

Middlewares for the Sharp Edges

The places you previously rigged headers by hand go in middlewares: a default browser-like user agent per spider, a rotating proxy selector from the proxies lesson, a custom retry on 429 before the built-in retry gives up. Because middlewares wrap every request and response, one small class reaches the entire crawl.

class UniqueProxyMiddleware:
    def process_request(self, request, spider):
        request.meta["proxy"] = spider.proxy_picker.next()

Develop in the Scrapy Shell

Tuning selectors against a live page is where the trial-and-error happens, and Scrapy gives you the debugger you wished you had in BeautifulSoup: scrapy shell https://example.com/products fetches the page and drops you into an interactive REPL with the parsed response already loaded. Test CSS and XPath selectors immediately, read response.xpath, check response.url after redirects, and confirm the exact .get() versus .getall() return before you ever save the spider:

scrapy shell https://example.com/products
>>> response.css(".product h3").get()
'<h3>Product 1</h3>'
>>> response.css(".product h3::text").getall()

What begins in the shell ends in a parse() callback, so the verify-the-selector loop costs seconds instead of run-and-die cycles. Treat the shell as the deep-work surface and the spider file as the frozen result of what you proved there.

Jobs That Pause

Scrapy's least-known gift is job persistence: run with -s JOBDIR=jobs/products and the scheduler, dedupe, and request queue persist across Ctrl-C and restart. That is the caching-and-resume lesson, free of charge, at the request level rather than the byte level. A production scraper that pauses, resumes, and reports its stats cleanly is the whole point of moving to a framework instead of a bespoke script.