The Crawl Is Not a One-Liner

A scraper that must be run by hand dies. The difference between a script and a system is that the system runs without you, notices when it fails, and tells someone. This is the operations lesson: put the pipeline on a schedule, give it idempotent, overlap-proof runs, and catch breakage the same way you catch a downed server — by monitoring freshness, not by waiting to be missed.

Cron and systemd Timers

Cron is the 90% answer: one line, per-user or system-wide, whatever cadence the data deserves.

# every day at 06:30
30 6 * * * cd /srv/scraper && ./venv/bin/python run.py --crawl nightly >> logs/crawl.log 2>&1

Never log to the void: >> crawl.log 2>&1 is the minimum, and the log is your postmortem state. For finer control (missed-run catchup, persistent logs, environment files), systemd timers are cron's replacement when you need OnCalendar= plus Persistent=true. Either way the pattern is the same: the schedule does not know about your crawler, it just fires the entry point.

Lock Against Overlapping Runs

A slow crawl that overruns its window collides with the next scheduled one, and two copies of an incremental scraper double the load against a target you promised to go easy on. Take the lock out of the file system:

flock -n /tmp/crawl.lock -c './venv/bin/python run.py --crawl nightly'

-n fails fast instead of queuing: if the previous run is still alive, the new invocation exits immediately, and the operator learns from the log why. Idempotency from the caching lesson then guarantees the skipped run costs nothing on the next tick.

A Health Table Is the Monitoring API

Give the crawler an opinion it can be asked for. Every run writes a status row — started, finished, items, errors, and the newest and oldest updated_at it produced — into your database.

CREATE TABLE crawl_runs (
    id INTEGER PRIMARY KEY,
    source TEXT,
    started_at TEXT,
    finished_at TEXT,
    items INTEGER,
    errors INTEGER
);

That table converts "is the site working" into a query. Your freshness gate is a single question: how old is the newest row I trust? Send any source whose MAX(updated_at) is older than its expected cadence into the alerting path.

SELECT source, COUNT(*) AS n, MAX(updated_at) AS newest
FROM items
GROUP BY source;

Alert on the Failure Modes That Matter

The alerts worth wiring are few and cheap. First, staleness: a source did not produce within its normal window — that is the drift detector for your whole pipeline. Second, volume collapse: today's item count under half of yesterday's rolling average implies a selector quietly returned nothing (the layout-drift warning from cleaning, automated). Third, error spike: runs with more than a threshold of retries/exceptions. Each of the three routes to a webhook or message with the source name and the number; a single tiny function covers all of them.

def alert(message):
    post_webhook("https://alerts.example.com/hook", {"text": message})

if newest_row_age_hours > expected_cadence_hours:
    alert("source products went stale")

Logs That Let You Debug Without a Terminal

Make the log the debugger: write structured lines with key=value for the things a postmortem needs (run id, url, status, retries, bytes) and rotate the files with logrotate. The reason is simple and applies to the whole discipline: the crawler you maintain has to be debuggable by the emails it sends and the logs it leaves, not by a person who happens to be watching it run at 06:30.