The Cryptographic Giveaway

When you write a scraper using requests or axios, you likely spoof the User-Agent header to pretend you are Google Chrome. However, before that User-Agent header is ever transmitted, your script must negotiate a secure HTTPS connection via the TLS Handshake.

This handshake is where 90% of basic bots are caught, thanks to JA3 Fingerprinting.

What is JA3?

During the TLS ClientHello packet, the client tells the server which cryptographic protocols it supports. Specifically, it sends: - The TLS Version - A list of supported Cipher Suites - A list of TLS Extensions - Supported Elliptic Curves

A standard Python environment (using OpenSSL) sends a very specific list of ciphers in a very specific order. A real Chrome browser sends a completely different list of ciphers, in a different order, utilizing modern extensions like GREASE (random reserved values designed to break middleboxes).

Cloudflare takes these handshake parameters and creates an MD5 hash—the JA3 Fingerprint. If your HTTP headers claim you are Chrome, but your JA3 fingerprint matches the known signature for Python urllib, the WAF instantly drops your connection with a 403 Forbidden.

The Successor: JA4

JA3 concatenates the raw TLS parameters and hashes them, which made it trivially spoofable by reordering ciphers. The newer JA4 fingerprint captures more discriminating information: it hashes the full ClientHello structure, including the order of extensions, the ALPN protocol lists, and the supported groups, broken into a human-readable string like t13d1516h2_8daaf6152771.... JA4 is significantly harder to forge by accident because there are far more independent ordering dimensions that must all match a real browser.

Case Study: Why the Order Matters

OpenSSL and BoringSSL (the library inside Chrome) negotiate connections differently. Even when you list identical cipher suites, the order in which they are presented, the padding, the presence or absence of the padding extension, and the supported signature algorithms all differ. Breaking any one of those details produces a unique JA4 that does not match any shipped browser, and a WAF with a modern TLS fingerprint database will spot it immediately.

HTTP/2 Pseudo-Headers

If you pass the TLS layer, modern WAFs inspect your HTTP/2 frames. HTTP/2 uses "pseudo-headers" (:method, :authority, :scheme, :path). - Chrome always sends them in the exact order: :method, :authority, :scheme, :path. - Firefox sends them in a slightly different order. - Standard HTTP libraries often scramble the order entirely.

Again, a mismatch between your stated User-Agent and your HTTP/2 frame structure results in an instant ban.

How to Spoof TLS in Python

You cannot fix this by simply changing headers in requests. You must fundamentally alter how the underlying C/Rust socket libraries negotiate the connection.

The modern standard for Python scraping is curl_cffi. It uses a compiled backend (borrowed from the Chrome source code) to flawlessly imitate the exact TLS handshakes and HTTP/2 behaviors of real browsers.

from curl_cffi import requests

# Notice we don't just pass a User-Agent string.
# We explicitly tell the library to impersonate the entire network stack of Chrome 120.
response = requests.get(
    "https://nowsecure.nl", # A Cloudflare test site
    impersonate="chrome120"
)

print(response.status_code) # 200 OK

curl_cffi ships a small catalogue of impersonate targets (Chrome, Safari, Firefox, and the current Edge revisions). Picking the newest browser version in the catalogue and re-pinning it when the WAF vendors update their detection is a maintenance task of its own.

The Limits of Faking

Impersonating a browser's TLS stack only gets you past the network layer. Every physical Chrome version has a fixed set of TLS behaviors; the WAF knows exactly what each version should look like. If you impersonate Chrome 120 while your User-Agent says Chrome 122, the version mismatch is detectable. Always keep your advertised browser version synchronized with the TLS stack you are actually impersonating.

By perfectly aligning your TLS signature, HTTP/2 frames, and User-Agent, you become statistically invisible to network-layer WAFs.