Building Resilient Scrapers

The internet is fundamentally chaotic. Servers go down, TCP connections drop, DNS lookups fail, and Web Application Firewalls arbitrarily throttle traffic.

If you write a scraper that assumes every HTTP request will succeed, it will crash during a long-running extraction job. Professional scraping infrastructure relies on robust error handling, explicit timeouts, and exponential backoff retry logic.

1. Enforcing Timeouts

By default, the Python requests library can hang indefinitely if the target server stops responding but doesn't close the socket. Always enforce a timeout.

import requests

try:
    # 3 seconds for connection, 10 seconds for reading data
    response = requests.get("https://example.com", timeout=(3, 10))
    response.raise_for_status() # Raises an exception for 4xx/5xx status codes
except requests.exceptions.Timeout:
    print("The request timed out. The server is unresponsive.")
except requests.exceptions.RequestException as e:
    print(f"A network error occurred: {e}")

2. The Tenacity Library for Automatic Retries

Writing manual retry loops with try/except and time.sleep() results in messy, unreadable code. The Tenacity library is the industry standard for retry logic in Python.

It allows you to declaratively define how and when a function should be retried using Python decorators.

import requests
from tenacity import retry, wait_exponential, stop_after_attempt, retry_if_exception_type

# Stop after 5 attempts
# Wait 2^x * 1 second between each retry (2s, 4s, 8s, 16s)
@retry(
    stop=stop_after_attempt(5),
    wait=wait_exponential(multiplier=1, min=2, max=30),
    retry=retry_if_exception_type(requests.exceptions.RequestException)
)
def fetch_data_robustly(url):
    print(f"Attempting to fetch {url}...")
    response = requests.get(url, timeout=10)

    # We explicitly raise HTTP errors so Tenacity catches them and retries
    if response.status_code in [429, 500, 502, 503, 504]:
        response.raise_for_status()

    return response.json()

3. Handling 429 Too Many Requests

If you receive a 429 status code, the server is explicitly telling you that you are sending requests too quickly. Respect this boundary. Look for the Retry-After header in the HTTP response—it tells you exactly how many seconds you must wait before the server will accept requests from your IP again. Integrate this specific delay into your retry logic.

4xx vs 5xx: Retry the Right Ones

The most valuable error-handling decision is knowing which errors deserve retries at all. Retry the transient world: 429 (slow down), and the 5xx family (the server is briefly failing or overloaded). Do not retry the permanent world: 400 (your request was malformed), 401/403 (you are not allowed in, repeatedly retrying exactly the credential that was refused is how accounts get banned), 404/410 (the content is gone and no retry recreates it). A retry policy that cannot tell 429 from 403 will retry its way into a longer ban.

import random
import time

def with_budget(fetch, url, attempts=4):
    last = None
    for i in range(attempts):
        try:
            resp = fetch(url)
            if resp.status_code < 500 or resp.status_code == 429:
                return resp
            last = resp
        except Exception as exc:
            last = exc
        time.sleep((2 ** i) + random.uniform(0, 1))
    raise RuntimeError(f"gave up on {url}: {last!r}")

Timeouts: Separate Connect From Read

A connection timeout and a read timeout measure different failures. The connection phase is "can I reach this host at all" — short, rarely needs help. The read phase is "is the server actually sending bytes" — long, and where slow targets live. timeout=(3, 10) means three seconds to connect, ten to read a full response, and it is the split default you should write everywhere, because a single over-long "read" timeout is how one dead server hangs your whole crawl.

A Failure Budget, Not an Endless Loop

Retries exist to ride out blips, not to brute-force a dead endpoint. Every retry policy should have a ceiling a human can see: a finite attempt count, exponential backoff with a cap, and jitter so ten retrying workers do not synchronize into a stampede. Log each failure as a first-class event (the monitoring lesson indexes them), and alert on the pattern, not the individual error. The scraper that "handles all exceptions" by retrying forever has done the opposite of error handling: it turned a recoverable outage into a scheduled DDoS.