See What Actually Traveled

Browser DevTools is enough for the first hour of investigating a target. Then you hit the wall it cannot cross: you need to replay a captured request with surgical edits, you need the headers and timing of hundreds of requests at once, or the target lives inside an app whose network traffic you cannot watch through DevTools at all. The professional toolkit for that wall is a HAR export for analysis and mitmproxy for interception. Both see the bytes that actually traveled, which is more than any debugger shows.

HAR: The Page's HTTP Logbook

A HAR (HTTP Archive) file is a JSON log of every request a page made and every response it received: URLs, methods, headers, cookies, POST bodies, timings, and status codes. DevTools exports it under Network → export HAR. Because it is JSON, it is scriptable — you can filter to "only API calls" or "only 5xx" in one pass, something the DevTools UI makes you click for.

import json

har = json.load(open("capture.har"))

for entry in har["log"]["entries"]:
    req = entry["request"]
    if req["method"] != "GET":
        print(req["method"], req["url"])
    resp = entry["response"]
    print("  ->", resp["status"], resp.get("content", {}).get("mimeType", ""))

The power of a HAR is being able to ask questions over the whole session, not just the request you happened to click on. "How many distinct API hosts did this page hit?" "Which POST body did the checkout use before the server rejected it?" HAR answers in seconds.

Replay From a HAR

Once you have filtered the API entries, replay them through httpx to test the hidden-API strategy before writing any scraping code. Copy the headers the page used, preserve the order where the target cares, and swap your own session in.

import httpx

client = httpx.Client(timeout=15.0)
for entry in har["log"]["entries"]:
    req = entry["request"]
    url = req["url"]
    if not url.startswith("https://api.example.com/"):
        continue
    headers = {h["name"]: h["value"] for h in req["headers"] if h["name"] not in ("host",)}
    params = [p["name"] for p in req.get("queryString", [])]
    print("replay:", req["method"], url, "params:", params)

Clipping the host header is deliberate: httpx sets it from the URL, and yes, some fragile targets choke on a duplicated host.

Mitmproxy: Rewrite the Traffic in the Middle

Where HAR reads, mitmproxy changes. Run as a proxy on your own machine (mitmproxy for the interactive TUI, mitmweb for a web UI, mitmdump for automation), point a browser or your scraper at it, and every request passes through your hands. Add-ons are plain Python functions on events:

from mitmproxy import http

def response(flow: http.HTTPFlow) -> None:
    content_type = flow.response.headers.get("content-type", "")
    if "application/json" in content_type:
        text = flow.response.text
        print(flow.request.host, flow.request.path.split("?")[0], len(text))

Pointing your scraper through a proxy in a debug harness is cheap and catches the whole class of "my scraper sends something different than the browser" bugs, because now you can diff the two byte-for-byte. Pair the captured output with the caching lesson's samples and you get a permanent regression test: the site changes its JSON shape, your proxy-based checksum snaps, and you know before the dataset does.

When the Browser Can't Be Your Eyes

The scenario DevTools cannot cover is an embedded webview or an app that bricks the diagnostics protocol. Run the app's traffic through mitmproxy in a port-forwarded phone or a container, and you recover exactly what the internal API expects. The capture, replay, and re-engineering skills are identical; only the container changed. That is the endpoint of this lesson: the tools that see the real bytes turn "the site that refuses to be scraped" back into "the API I can now replay politely."