Operational Security for Scrapers

Bypassing a WAF on a single request is easy. Bypassing a WAF for a sustained 10-million-row extraction job over three weeks requires meticulous Operational Security (OpSec) and behavioral discipline.

WAFs utilize machine learning models that analyze aggregated traffic patterns. If you act like a machine, you will be banned like a machine.

1. Header Perfection and Ordering

We know that spoofing the User-Agent is mandatory. But you must spoof the entire header block flawlessly. If your User-Agent claims to be Chrome 120 on macOS, but you send the Accept-Encoding: gzip, deflate header instead of Chrome's modern Accept-Encoding: gzip, deflate, br, zstd, you create an anomaly score. Furthermore, Header Ordering matters. Python's requests alphabetizes headers. Chrome sends them in a specific, rigid, non-alphabetical order. WAFs check this. Use libraries like curl_cffi that handle order automatically.

2. The Illusion of Human Timing

Machines operate at the speed of the CPU. If you scrape a site with a hardcoded time.sleep(2) between requests, the WAF analyzes the timestamp deltas: 2.001s, 2.000s, 2.002s. Humans do not click links with millisecond perfection.

Implement Gaussian Jitter: Use a normal distribution to randomize your delays. Most requests will take around 3 seconds, but occasionally one will take 1 second, and another will take 7 seconds (simulating a user getting distracted).

import time
import random

def human_delay(mu=3.0, sigma=1.0):
    # Generates a delay following a bell curve, clamped between 1 and 8 seconds
    delay = max(1.0, min(8.0, random.gauss(mu, sigma)))
    time.sleep(delay)

3. Simulating Human Motor Functions

If you are using Playwright/Selenium, never use the default .click(x, y) method. It teleports the mouse cursor instantly to the target coordinate. Datadome records mouse telemetry and will ban you for teleporting.

Use specialized libraries that generate Bézier curves to smoothly animate the mouse cursor across the screen, overshooting slightly, and correcting back to the target button—exactly how a human hand behaves.

4. The "Ghost Town" Approach

If you need to scrape 100,000 pages, do not do it from a single IP at 50 requests a second. Use a massive residential proxy pool and deploy a distributed fleet of slow scrapers. If you have 500 IPs, and each IP only makes 1 request every 60 seconds, your aggregate throughput is extremely high, but from the WAF's perspective, they just see a bunch of distinct, incredibly slow, perfectly normal human users.

5. Session Shape

Match your session shape to the workflow shape. A logged-in session that makes one call to a search endpoint, then 40 calls to detail pages, then logs out, looks human. A session that makes 400 calls to a single unauthenticated endpoint with no reads in between looks machine-like. Vary the shape naturally: interleave occasional home-page and "about" round trips, and carry cookies for the entire workflow instead of re-creating sessions per request.

6. Avoid Honeypot Traps

Many WAFs embed invisible honeypots: links hidden with display: none or off-screen positions that real users can never find or click. A scraper that follows every link in the DOM will gleefully navigate into the honeypot and get flagged. Only traverse links on visible, interactive paths, and verify the target is not marked with the standard honeypot attributes.

7. Account for TLS Session Reuse

Real users browse a site with one TLS connection kept alive across many requests. If your scraper opens a fresh TLS handshake per request, the connection-per-request ratio becomes a statistical outlier even when every packet looks perfect. Keep a persistent pooled connection (and, where possible, an HTTP/2 session) for the lifetime of a crawl batch.