Data That Arrives, Not Data You Fetch

Some data has no pages at all. A market feed, a live auction, a streaming log viewer, a chat: the server pushes frames down a persistent connection and the page never performs a single load. Scraping these endpoints is not HTTP — it is subscribing. The mechanism is WebSocket, a long-lived TCP socket with an HTTP upgrade handshake, and the skill is split between speaking the protocol and keeping the feed alive.

Spot the Handshake

In DevTools, the Network tab shows WebSocket connections under the WS filter, and the Messages side-panel lets you read the frames. The handshake happens once: a normal HTTP request with Upgrade: websocket, after which the same TCP connection carries frames in both directions. The URL (wss://.../ws), the query parameters, and any Authorization header or cookie from the handshake are your entire access contract.

Subscribe in Python

The websocket-client library is the pragmatic synchronous choice for a scraper: create_connection returns a socket, send writes a frame, recv blocks until one arrives.

import json
from websocket import create_connection

ws = create_connection("wss://stream.example.com/ws?token=abc")
ws.send(json.dumps({"action": "subscribe", "channel": "trades"}))

for _ in range(50):
    frame = ws.recv()
    msg = json.loads(frame)
    print(msg)

Most feeds ask you to subscribe to channels by name. The subscription message — channel, symbols, filters — is the equivalent of the query selector: get it exactly right or you receive nothing but heartbeats.

Stay Alive Through Protocol and Silence

Two things kill a subscription: the server's ping/pong and the connection's idle timeout. Many servers silently drop a socket that exchanges no frames in N seconds; a scraper must therefore keep the link warm and reconnect when it dies. The loop-with-reconnect is the canonical shape:

import json
import time
from websocket import create_connection

def consume(url, since):
    while True:
        try:
            ws = create_connection(url, timeout=20)
            ws.send(json.dumps({"action": "subscribe", "channel": "trades"}))
            while True:
                frame = ws.recv()
                if frame is None:
                    break
                yield json.loads(frame)
        except Exception as exc:
            print("reconnecting:", exc)
            time.sleep(5)

Respond to server pings ({"type":"ping"}) by echoing the pong or answer {"type":"pong"} where the contract demands it, and flush partial buffers between reconnects. A feed consumer that reconnects cleanly is a feed consumer that does not miss a week of data because of one DHCP renew.

Intercept an App's WebSocket With a Browser

Not every feed lets a third party subscribe. For an app-internal stream, turn the browser-automation lesson to your advantage: inject an init script that watches WebSocket.prototype.send and collects what the app pushes. Then a single page visit records the whole feed.

page.add_init_script(
    "window.__frames = [];"
    "var origSend = WebSocket.prototype.send;"
    "WebSocket.prototype.send = function (data) {"
    "  window.__frames.push(data);"
    "  return origSend.apply(this, arguments);"
    "};"
)

Read page.evaluate("window.__frames") periodically. This is the capture-middle-path applied to the stream: you never reproduce the feed logic, you just record the app doing what it already does.

The Engineering Floor

Real-time data deserves real-time handling: dedupe on a stable message id (feeds redeliver), store frames in append-only journal lines (the caching lesson's journaling habit), and batch-write to storage in small windows instead of one transaction per tick — at millions of events a day the per-event write is the bottleneck. Keep the consumer's only job small: decode, dedupe, persist. Anything else running in that loop is how a live feed starts eating a weekend.