The Two Faces of Authentication

Almost every target worth scraping puts at least part of its data behind a login. Servers authenticate you two ways. The older way is a server session: you submit credentials, the server stores a session on its side and hands you an opaque session cookie (often named JSESSIONID, sessionid, or PHPSESSID), and every later request is recognized by that cookie. The newer way is stateless tokens: you submit credentials to a token endpoint, and it returns a signed access token (usually a JWT) that the server verifies on every request without storing anything.

The two styles demand different code. Cookie sessions mean managing a cookie jar. Token APIs mean watching for expirations and refreshing. Many real sites are hybrids, and the discipline that keeps all of them running is the same: understand what authenticates you, keep it valid for the whole run, and persist it so the next run starts authenticated.

Logging In the Cookie Way, With CSRF

Modern cookie logins add one wrinkle: a CSRF token embedded in the login form. The login endpoint rejects submissions without the exact token that belongs to the current session. The flow is therefore always three steps, never two: open the login page with a Client, read the token out of the HTML, and submit it together with the credentials using the same client.

import httpx
import json

client = httpx.Client(base_url="https://shop.example.com", timeout=15.0)

# 1. GET the login page to obtain the session cookie + CSRF token
html = client.get("/login").text
csrf = html.split('name="csrf" value="')[1].split('"')[0]

# 2. POST credentials with the token, same client keeps the session cookie
login = client.post(
    "/login",
    data={"email": "you@example.com", "password": "secret", "_csrf": csrf},
)
login.raise_for_status()

The Client is the whole trick: it stores the Set-Cookie headers automatically, so the follow-up /account, /orders, and /api/me requests all arrive authenticated. Keep exactly one client per identity.

Persist the Session Across Runs

Logging in on every startup is slow and increasingly triggers "suspicious login" checks. Export the authenticated session to disk, and on the next run, restore it in one line.

jar = dict(client.cookies)
with open("session.json", "w") as f:
    json.dump(jar, f)

# next day, no login required:
jar = json.load(open("session.json"))
client.cookies.update(jar)

Sessions expire, so combine restoration with a cheap probe: hit an endpoint that only works when logged in, and re-login (fresh CSRF flow) only when it returns a redirect to the login page or a 401.

Access Tokens and Refreshing

Token APIs normally hand you two tokens. The access token is short-lived (minutes to hours) and goes in an Authorization: Bearer header. The refresh token is long-lived and is only ever exchanged for a new access token. Code the refresh at exactly one place, in a small wrapper, not scattered in every caller.

def api_get(client, url, tokens):
    resp = client.get(
        url,
        headers={"Authorization": "Bearer " + tokens["access"]},
    )
    if resp.status_code == 401:
        refresh = client.post(
            tokens["token_url"],
            json={"grant_type": "refresh_token",
                  "refresh_token": tokens["refresh"]},
        )
        refresh.raise_for_status()
        tokens["access"] = refresh.json()["access_token"]
        resp = client.get(
            url,
            headers={"Authorization": "Bearer " + tokens["access"]},
        )
    return resp

The refresh wrapper works because the failure is detectable and the retry is identical to the original. If the refresh itself returns 401, the refresh token is dead: fall back to a full credential login with the CSRF flow above.

Log In With a Browser When There Is No Clean Endpoint

Some apps generate their sign-in state with JavaScript, or use an SSO provider where the login flow is web-only. This is the one case to leapfrog straight to browser automation: run the login once in Playwright, then export the resulting cookies into your httpx session.

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto("https://app.example.com/login")
    page.locator('input[name="email"]').fill("you@example.com")
    page.locator('input[name="password"]').fill("secret")
    page.locator('button[type="submit"]').click()
    page.locator("#dashboard").wait_for()
    cookies = browser.contexts[0].cookies()
    browser.close()

import httpx
client = httpx.Client(headers={"Cookie": "; ".join(
    c["name"] + "=" + c["value"] for c in cookies
)})

Once exported, the browser session cookie keeps working in plain HTTP for as long as it lives. This is the practical way to authenticate against login walls you would otherwise be unable to automate, and it is a pattern worth keeping for any stubborn SSO. Bundle the cookie export into the same persistence file from the previous section so re-login becomes an explicit, rare event.