Scrape the Data, Not the DOM

Modern frontends are rarely the thing that actually serves the data. A React or Vue page is usually an empty shell. The browser runs a bundle of JavaScript, and that script spends its whole life making fetch() and XHR calls to an internal API that returns clean JSON. Every row you see on screen traveled over the network as a structured payload.

If you can find that API and call it directly, you skip a category of problems entirely. No CSS selectors, no waiting for animations, no rendering engine. You get exactly the same records the site itself gets, in a shape you can load straight into a database. The skill of turning a website into its API is the second most valuable thing you can learn in scraping, after knowing HTTP itself.

Why the API Beats the DOM

The DOM is a snapshot of a page at one instant. It is full of noise: headers, footers, ads, tracking scripts, hidden elements, dehydrated state in <script> tags. To get one price out of it you write a selector, wait for the layout to change, and rewrite the selector. The internal API, on the other hand, returns a contract. The shape barely changes, the payload is smaller, and the fetch is dozens of times faster than rendering a browser.

The performance gap is the real point. A requests-style call to a JSON API costs tens of milliseconds and a few kilobytes. A headless browser costs hundreds of megabytes of RAM and several seconds per page. If your crawl runs over a million pages, that difference is the difference between finishing in a day and running for a month.

Step 1: Capture the Network Traffic

Open DevTools in any Chromium browser and click the Network tab. Reload the target page. Look through the request list for XHR and Fetch calls. The interesting requests usually have a few tells:

  • The response is JSON, not HTML.
  • The URL contains words like api, api/v2, graphql, search, listing, or a resource name like products.
  • The URL carries query parameters that map to what is on screen (page number, category id, search term).

Right-click a promising request, choose Copy and then Copy as cURL. That gives you the full request: URL, method, headers, and body. Do not copy it into a shell and run it verbatim. Parse it for the three parts that matter: the endpoint, the parameters, and the headers the server actually cares about.

Step 2: Replay the Request Outside the Browser

Using httpx in Python, replay the request. Start minimal. You almost never need the fifty headers the browser sends. Most APIs care about a browser-like User-Agent, Accept, and possibly a token or cookie.

import httpx

url = "https://store.example.com/api/products"
params = {"category": "keyboards", "page": 1, "per_page": 50}
headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
    "AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0 Safari/537.36",
    "Accept": "application/json",
}

with httpx.Client(headers=headers, timeout=10.0) as client:
    resp = client.get(url, params=params)
    resp.raise_for_status()
    data = resp.json()

print(data.get("total"))
print(data["results"][0])

If the response comes back, you now have the site's data pipeline without a browser. Replay a few requests and confirm the parameters change the output the way the UI changes when you click through the real site. When a new column or filter appears in the interface, watch the network tab again and a new parameter will show up in the request. That is the whole loop.

Step 3: Walk Pagination Through the API

The API paginates by one of two styles. An offset style uses page, offset, or start and increments until the list is empty. A cursor style returns an opaque next_cursor token, and you pass that exact token back in until the server returns a null cursor. Cursors are strictly better because they snapshot the list at the time of the first request, so items added mid-crawl do not shift results around.

import httpx

def save(record):
    print("storing:", record["id"])

client = httpx.Client(timeout=10.0)
url = "https://store.example.com/api/products"
cursor = None

while True:
    params = {"per_page": 100}
    if cursor:
        params["cursor"] = cursor
    resp = client.get(url, params=params)
    resp.raise_for_status()
    body = resp.json()

    for item in body["results"]:
        save(item)

    cursor = body.get("next_cursor")
    if not cursor:
        break

With an offset API, replace the cursor logic with a page counter and stop when a page returns an empty list. Guard against a bug where the site ignores the offset and keeps returning the same page forever: track a hash of the first item on each page and break if you see a repeat.

Step 4: Handle Auth Headers and Tokens

If the API is behind a login, the browser stores a session cookie or a bearer token after you sign in. With httpx, a plain client keeps cookies across requests, so you can log in once and stay authenticated.

import httpx

with httpx.Client(timeout=10.0) as client:
    login = client.post(
        "https://store.example.com/api/login",
        json={"email": "you@example.com", "password": "secret"},
    )
    login.raise_for_status()

    data = client.get(
        "https://store.example.com/api/orders",
        headers={"Accept": "application/json"},
    ).json()

For token-based APIs, grab the token from DevTools (it is usually in an Authorization: Bearer ... header on API requests, or stored in localStorage) and attach it yourself:

headers = {"Authorization": "Bearer eyJhbGciOi..."}
resp = client.get("https://api.example.com/v2/orders", headers=headers)

When the API Is Locked Down

Some sites sign every request, embed the client timestamp in a hash, or require a token minted by an obfuscated script. Reversing that signature system is a treadmill: the site redeploys and your solver breaks. This is the moment to fall back to the headless browser from the previous lesson and let a real Chromium execute the signing code for you. Read the lesson on caching before you do anything else, because a browser session is expensive and you do not want to pay for it twice for the same page.