Pages That Are Never Really Done

Modern feeds do not page. They scroll. A page "loads" with a skeleton and fills in as you move: infinite scroll appends items as you approach the bottom, images lazy-load when they enter the viewport, and virtualization (react-window and friends) renders only the rows you can see while discarding the rest from the DOM. The naive scraper reads the initial HTML, sees three items, and gives up. The correct scraper answers one question first: where does the data actually come from?

Anatomy of the Skeleton Page

Open the Network tab and scroll. The pattern you will see repeats with boring regularity: the first request is a small HTML shell, then a JavaScript library boots, then XHR/fetch requests to something like api/search?page=N or feed/load-more return JSON blocks, and the app injects them into the DOM. That means the site has already solved your pagination for you with a JSON API. Use the hidden-API lesson: replay the page=N (or cursor) requests directly. This is the fastest, most reliable route, and it sidesteps the rendering engine entirely.

import httpx

client = httpx.Client(timeout=15.0)
page = 0
while True:
    data = client.get(
        "https://example.com/api/search",
        params={"q": "keyboards", "page": page},
        headers={"Accept": "application/json"},
    ).json()
    items = data["results"]
    if not items:
        break
    for item in items:
        save(item)
    page += 1

When There Is No API: Drive the Scroll

Not every lazy load maps to a JSON endpoint. Sometimes the endpoint is signed, virtualized to the point of uselessness, or the renderer derives rows client-side. Then you scroll a real browser. The loop is: scroll to the bottom, wait for the next batch, check whether the DOM actually grew, and stop when it stops.

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto("https://example.com/feed", wait_until="domcontentloaded")

    previous = 0
    while True:
        page.mouse.wheel(0, 1600)
        page.wait_for_timeout(800)
        count = page.locator(".feed-item").count()
        if count == previous:
            break
        previous = count

    titles = page.locator(".feed-item h3").all_inner_texts()
    print("collected", len(titles))

Three details matter here. First, mouse.wheel drives the scroll that lazy loaders actually watch (a window.scrollTo jump is fine too, but wheel is the closest to a human). Second, the count == previous comparison is the stop condition, and it is also your drift detector: if the site starts scrolling endlessly, the loop tells you instead of hanging. Third, give each batch a real wait — lazy loaders debounce on scroll events, and wait_for_timeout(800) is the honest price of patience when triggers are event-based, in sharp contrast to the waiting discipline in browser-automation, which avoids fixed sleeps for presence, not for settling.

Virtualization: The DOM Stops Growing

The hardest variant, virtualization, never grows the DOM past the visible window: scroll far enough and those rows are reused for new content. The count == previous trick never fires; the loop scrolls forever. The end-of-list is then revealed by geometry, not by element count.

def near_bottom():
    return page.evaluate(
        "document.documentElement.scrollHeight"
        " - window.scrollY - window.innerHeight < 50"
    )

while not near_bottom():
    page.mouse.wheel(0, 1600)
    page.wait_for_timeout(700)

And because virtualization means the DOM eventually forgets items, you must extract as you go, collecting each batch before it scrolls away.

The Hidden Gold: JavaScript State

One more source is inside the page itself. Single-page apps routinely stash their full dataset in globals like window.__INITIAL_STATE__, window.__PRELOADED__, or data- attributes on the shell element before rendering. When the feed is virtualized and the API is signed, read the app's own memory:

state = page.evaluate("window.__INITIAL_STATE__ || null")
if state:
    import json
    records = json.loads(state)

Check for these globals before writing a single scroll loop. They are one line of code away and they return the entire dataset in one shot. The scroll loop is then just the fallback for when the app refuses to expose its state.