Trust Nothing, Verify Everything

A scraper has one job: it produces records as clean as someone typed them by hand. Raw HTML hands you strings with currency symbols, stray spaces, HTML entities, and missing attributes. If those strings flow straight into a database, the whole dataset becomes a liability at analysis time. Cleaning and validation are not the boring bit you can skip. They are the difference between a dataset you trust and a dataset you quietly avoid.

Normalize Before You Store

Decide on canonical forms and convert every scraped value into one. Prices become numbers, dates become ISO 8601 strings, ids become stripped text, booleans become actual booleans. Do the conversion closest to the parse, once, so every downstream consumer sees the same shape.

import re
from datetime import datetime

def parse_price(raw):
    cleaned = re.sub(r"[^0-9.]", "", raw or "")
    if not cleaned or cleaned.count(".") > 1:
        return None
    return float(cleaned)

def parse_date(raw):
    for fmt in ("%Y-%m-%d", "%d %b %Y", "%b %d, %Y"):
        try:
            return datetime.strptime(raw.strip(), fmt).date().isoformat()
        except ValueError:
            continue
    return None

Note the failure mode: a value that cannot be parsed becomes None, not a silent string. You want to see the gap between how many records you scraped and how many parsed cleanly.

Deduplicate on a Stable Key

Pages repeat records. Ads embed the same product on ten listing pages, and the same article shows up under two categories. Deduplicate on a business key that is independent of the page: a SKU, an external id, a canonical URL. Push the constraint into the database so duplicates cannot slip in even when your own code forgets to check.

CREATE TABLE products (
    sku        TEXT PRIMARY KEY,
    title      TEXT NOT NULL,
    price      REAL,
    fetched_at TEXT NOT NULL
);

INSERT INTO products (sku, title, price, fetched_at)
VALUES (?, ?, ?, ?)
ON CONFLICT(sku) DO UPDATE SET
    title = excluded.title,
    price = excluded.price,
    fetched_at = excluded.fetched_at;

An upsert like this also solves an honesty problem: you can re-scrape a product and update its price, and the row is still one row.

Validate the Shape of Every Record

After normalization, assert invariants before saving. The checks do not need to be clever; they need to exist. Require the fields you will query, require types, and sanity-check ranges. A price that sorts below zero or a date in the past is a red flag worth logging.

REQUIRED = {"sku", "title", "price"}

def valid_record(record):
    if not REQUIRED.issubset(set(record)):
        return False
    if not isinstance(record["price"], float):
        return False
    if record["price"] < 0:
        return False
    return True

Detect Layout Drift Before It Silences You

The most expensive failure in scraping is the one that parses but returns empty fields. The site changes a class name, your selector returns nothing, and the pipeline stores hundreds of records of None. Catch this with a shape check: count how many records today have every required field populated, and alert when the ratio drops.

from collections import Counter

completeness = Counter()
for record in records:
    completeness[bool(record.get("price"))] += 1

pct = completeness[True] / max(1, len(records))
if pct < 0.95:
    raise RuntimeError(f"completeness dropped to {pct:.0%}, parser drifted")

The alert is the point. A failed scrape you hear about gives you a ten minute fix. A scrape that quietly returns garbage for a day takes all day to redo.

Keep Samples Instead of Trusting Memory

When a target changes, you want the old markup to debug against, not a memory of it. Keep a running handful of raw HTML samples per selector path, so when completeness drops you can diff the old sample against the new page and see the exact class rename or attribute move. This is the cheapest maintenance habit in the whole job, and it pays for itself the first time a layout changes.