One Endpoint, Every Query

GraphQL is not a new protocol, it is a different contract. Instead of many REST resources and many endpoints, an app publishes one endpoint (almost always /graphql) that accepts a POST whose JSON body names the data you want. The server walks your query against its schema, executes exactly what you asked, and returns a nested JSON document matching your shape. Your scraper gets precisely the fields it requested, no oversized HTML payload, no undocumented resource URLs to discover.

This is good news for scraping: once you know the schema, every screen of the site is reachable through the same door, and you can request several pages of real data in one network call through cursor pagination. The skills are finding the endpoint, learning the schema, and driving cursor loops.

Finding the Endpoint

Open the Network tab and reload the app. Filter by Fetch/XHR and look for the request whose URL ends in graphql, or whose path is /api with a body containing "query" and "variables". Some apps split traffic across graphql and REST; the pages that feel "heavy JS" are almost always GraphQL-backed. Right-click the request, Copy as cURL, and note the operation name and the exact variables the app sent. That captured request becomes your first honest query.

Learn the Schema With Introspection

Unless the server disabled it, GraphQL tells you everything about itself. Send the introspection query and the response is a machine-readable menu of every type and field.

import httpx

client = httpx.Client(base_url="https://api.example.com", timeout=15.0)
introspection = {
    "query": (
        "query IntrospectionQuery { __schema { "
        "queryType { fields { name } } } }"
    )
}
resp = client.post("/graphql", json=introspection)
schema = resp.json()
for field in schema["data"]["__schema"]["queryType"]["fields"]:
    print(field["name"])

Introspection is often switched off in production. Then your schema source is the DevTools capture itself: every field you see in captured queries is field-blessed by the app, so rewrite captured queries rather than inventing fields from memory.

Persisted Queries: When There Is No Full Query

Compiler-style GraphQL clients (Relay, Apollo with persisted queries) do not always send the whole query string. They send a short hash in extensions.persistedQuery and the server looks up the full operation on its side. The hashes are findable: they ship in the app's JavaScript bundle (often a file of operation maps) or in a manifest under a path like /persisted_queries. One practical route is simpler than reading the bundle: watch the Network tab, find the captured request, and replay the exact extensions object. Most servers accept the app's own hash without the full string.

body = {
    "extensions": {
        "persistedQuery": {
            "version": 1,
            "sha256Hash": "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef",
        }
    },
    "variables": {"first": 50, "cursor": None},
}
data = client.post("/graphql", json=body).json()

Cursor Pagination Loops

GraphQL collections paginate with cursors, rarely with page numbers. The connection pattern returns a pageInfo block with hasNextPage and endCursor; you feed endCursor back in the next request until the flag flips.

query = (
    "query Items($cursor: String) {"
    '  items(first: 100, after: $cursor) {'
    "    pageInfo { hasNextPage endCursor }"
    "    nodes { id title price }"
    "  }"
    "}"
)

cursor = None
while True:
    body = client.post("/graphql", json={
        "query": query,
        "variables": {"cursor": cursor},
    }).json()
    conn = body["data"]["items"]
    for node in conn["nodes"]:
        save(node)
    if not conn["pageInfo"]["hasNextPage"]:
        break
    cursor = conn["pageInfo"]["endCursor"]

The after parameter takes the exact opaque string the server returned. Never convert cursors to integers; some servers base64-encode an offset, but treating them as tokens is the contract.

Cost, Aliasing, and Politeness

GraphQL servers evaluate the cost of each query from its depth and breadth. A "cheap-looking" recursive query that nests 50 levels deep or requests an enormous list can be rejected or throttled because the server priced it beforehand. Balance each query to the smallest shape that answers your need, one page at a time, and respect extensions: { throttle }-style hints if the server reports usage. Because one body can carry multiple independent operations, be conservative when combining them in a single call: a partial failure of one operation fails them as a unit. When in doubt, keep one operation per request and rely on the cursor loop just shown, which is exactly as fast and far easier to reason about.