When Only a Browser Will Do

Every earlier lesson steered you toward the fastest path: plain HTTP, and wherever possible the hidden internal API. But some targets make that impossible. The page is rendered entirely by JavaScript, the API is signed, or the site runs a proof-of-work challenge only a browser engine can solve. For those, you stop impersonating a browser and start driving one. Playwright is the tool: it controls a real Chromium, Safari, or Firefox over a debugging protocol, so the server sees an authentic browser while your code decides where it goes and what it reads.

The moment a request goes through a real browser, most fingerprint objections vanish. The TLS stack is genuine, the headers come out in the right order, the JavaScript executes, and the site's own code signs its own requests for you. You trade that authenticity for speed: a browser costs hundreds of megabytes of RAM and seconds per page. Automation is the blunt instrument of scraping, and you should treat it as the last resort this course positions it as.

Launch, Contexts, Pages

A Browser is a full engine with one tab per Page. Between those two sits the piece you will actually care about: the Context, an isolated profile with its own cookies, localStorage, service workers, and cache. Contexts are cheap and independent, so one browser process can host as many contexts as you want parallel crawls, each with a different identity.

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    context = browser.new_context(
        user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
        "AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126 Safari/537.36",
        locale="en-US",
        viewport={"width": 1440, "height": 900},
    )
    page = context.new_page()
    page.goto("https://example.com/search", wait_until="domcontentloaded")
    browser.close()

headless=True is the default and fine for most scraping. Keep headless=False only to watch a session during development. Communicate identity through the context, not through ad-hoc headers.

Locators Replace Manual Waiting

The classic brittle script does time.sleep(3) and hopes the data loaded. Playwright's Locator is self-waiting: every operation polls until the element is found or a timeout passes, so your code describes the target instead of the timing. A search flow becomes:

page.locator('input[name="q"]').fill("keyboards")
page.locator('input[name="q"]').press("Enter")
page.locator(".results .card").first.wait_for(timeout=10000)

cards = page.locator(".results .card")
print("cards:", cards.count())
for i in range(cards.count()):
    print(cards.nth(i).locator("h3").inner_text())

Only reach for explicit expect() or wait_for() when the next action logically depends on the previous one. Prefer page.wait_for_load_state("networkidle") sparingly, because busy pages never go idle and you will hang waiting for a timer that never ends.

Actions and Extraction

Locators cover the interactive surface: .click(), .fill(), .select_option(), .press(), .check(), .hover(). Playwright checks actionability before acting, which means it will scroll into view, wait, and retry until an element is really clickable. When the data you want sits inside an expression rather than a simple element, run JavaScript in the page:

texts = page.evaluate(
    "[...document.querySelectorAll('.item h3')].map(el => el.textContent)"
)

For server-rendered blocks, .inner_text() and .get_attribute() are enough. For grids built by frontend frameworks, evaluate with a selector + text extraction is usually both simpler and faster than chaining locators.

Shadow DOM and Iframes

Shadow DOM is how modern sites wall off their widgets. Playwright pierces open shadow roots automatically, so page.locator("luxon-widget button.confirm") just works even though the button lives inside a shadow root. Closed shadow roots are deliberately hidden; you can still reach them with element handles in evaluate, but treat that as a hostile-layout fallback. Iframes are a separate document tree: target them explicitly.

frame = page.frame_locator("#checkout-frame")
frame.locator('input[name="card"]').fill("4242 4242 4242 4242")

Watch the Traffic Through the Page Eyes

Because the page owns its network stack, its own requests are the honest ones. Catch every API response the page receives:

with page.expect_response(
    lambda r: "api/items" in r.url and r.request.method == "GET"
) as resp_info:
    page.locator("#next-page").click()

resp = resp_info.value
payload = resp.json()
print(len(payload["results"]))

This combines the browserless spirit of the earlier API lesson with the browser's ability to authenticate and sign on your behalf. Use page.route("**/api/**", lambda route: route.abort()) to block heavy analytics that slow the page down, and accept the query parameters the page itself chose to send.

The Cost of Running a Browser

A browser is a process with a VM, a layout engine, and a JavaScript runtime. Keep one long-lived browser instead of launching per page, batch work into contexts, and close contexts the moment a phase finishes so the memory is reclaimed. Never run a browser in the same process as your main worker loop; isolate it. And keep the order of operations in this course: the API path first, the browser as the sharp tool for the sites that demand it.