The Signature Problem

The last anti-bot trick in this course is the most annoying because it needs the most thought. Instead of issuing challenges, a site may sign every API request with a header derived from the request itself, the current time, and a secret key buried in its JavaScript. The server rejects any request whose signature does not match the inputs it sees. Example headers you will meet: X-Sign, X-Ts, X-Nonce, or a x-time/x-hash pair. The site is not hiding the data, it is making the data impossible to request without computing the same credential the app computes.

You have three ways forward: work out the signing algorithm and replicate it in Python, let a browser compute it for you and capture the result, or drop the target. This lesson is about the first two, and about knowing when each one (and the third) is correct.

The Observational Attack

Before reading a single line of JavaScript, gather data. Capture ten requests to the same endpoint with the network tab, spaced a few seconds apart, and diff them. You will usually find:

  • A timestamp header that changes on every request.
  • One or two hash headers that also change.
  • A constant header (or prefix) that stays identical.

That is the whole signing model compressed into three fields: a moving timestamp, a signature over the timestamp plus request, and a client identifier. Most homegrown signers are HMAC-SHA256 over a canonical string. Guess a canonical form, hash, and test it against your captured samples until it matches.

import hashlib
import hmac
import time

ts = str(int(time.time()))
msg = "GET|/api/items?page=1|" + ts
sig = hmac.new(b"key-from-bundle", msg.encode(), hashlib.sha256).hexdigest()

headers = {"X-Ts": ts, "X-Sign": sig}
resp = client.get("/api/items", params={"page": 1}, headers=headers)

The canonical order matters: method, path, query, timestamp is the common template, but some signers hash sorted query keys, some include a body hash, some bind the client id. Derive the exact template by replaying your captured inputs and comparing hex digests.

Reading the Minified JavaScript

When guessing fails, the key genuinely lives in the bundle. Open DevTools Sources, pretty-print the main script (the {} button), and press Ctrl+Shift+F to search for the literal string of the header you saw, say X-Sign. You land on the code that builds it. Minified names are impenetrable but the logic is not: look for a createHmac or hmac call, a sha256, a timestamp from Date.now(), and string concatenation of the fields. Step through with a breakpoint on the fetch wrapper while the page makes a real request, and inspect the arguments to the signing function. You do not need to understand the whole file, only the few lines that produce the value you must reproduce.

The Capture Middle Path

Replication is fragile the moment the signer includes a random nonce, an encrypted token from an obfuscated function, or a key that rotates weekly. Before you commit days to a virtual machine of decoding, consider letting the browser do the work and capturing its own signed requests with Playwright. The page authenticates, signs, and paginates itself; you observe the responses and keep the exact data flows you laid out in the browser-automation lesson.

with page.expect_response(
    lambda r: "api/items" in r.url
) as resp_info:
    page.locator("#next-page").click()

resp = resp_info.value
payload = resp.json()

This is the engineering equivalent of the observational attack: instead of reproducing the signer, you record its output. It is not free, it still costs a browser and the site's interaction budget, but it never breaks when the site reshuffles its bundler. Only the site rewriting its entire signing scheme can break it, and that is the signal to re-evaluate the target.

Knowing When to Stop

Signing schemes are a treadmill. Keys rotate, obfuscation layers stack, signatures bind to session state, and some defenses use server-issued nonces that make pure client-side replication impossible by design. The professional move is a budget: if the capture middle path works, use it. If the site rotates its obfuscation faster than you can ship, the internal-API strategy is gone, and you are paying the full browser price every crawl, then the economically honest answer is to stop scraping that property or negotiate. A dataset that costs you a flow, a signature, and a browser to extract is a dataset whose extraction cost you should have priced before you started. The earlier lesson on legal considerations is where that conversation belongs; this lesson is where you realize the bill.