Crawling Across Multiple Pages

Rarely does a website display all its data on a single page. Products, articles, and search results are almost always paginated. To build a comprehensive dataset, your scraper must be able to detect pagination and autonomously navigate through all available pages.

There are three main types of pagination you will encounter in the wild.

1. URL Parameter Pagination (The Easiest)

Many sites use simple query parameters like ?page=2 or offset parameters like ?start=20&limit=20.

For these, you simply wrap your scraping logic in a while loop, incrementing the page number until you hit a page with no results or a 404 error.

import requests
from bs4 import BeautifulSoup

base_url = "https://example.com/products"
page = 1
all_products = []

while True:
    print(f"Scraping page {page}...")
    response = requests.get(f"{base_url}?page={page}")
    soup = BeautifulSoup(response.text, 'html.parser')

    items = soup.select('.product-item')
    if not items:
        print("No more products found. Exiting.")
        break # Exit loop when pages are empty

    for item in items:
        all_products.append(item.text.strip())

    page += 1

2. "Next Page" Button Following

Instead of guessing URL parameters, a more robust method is to actively look for the "Next Page" link in the HTML DOM. This works beautifully for sites with complex URL structures or hash-based routing.

You extract the href attribute from the <a class="next-button"> element. If the element exists, you update your URL and fetch again. If it is disabled or missing, you have reached the end.

3. API Cursor Pagination (Infinite Scroll)

Modern React/Vue applications often implement "Infinite Scroll." When the user scrolls down, the frontend makes an asynchronous XHR request to an API endpoint.

These APIs usually use Cursor-based pagination. The JSON response will include a next_cursor token. To get the next batch of results, you must include that exact cursor token in your next API request.

To scrape this, do not use Selenium to scroll. Instead, monitor your browser's Network tab, find the backend API endpoint, and replicate the cursor-passing logic directly in Python using requests. It is exponentially faster and less prone to breaking.

Detecting Pagination Blind

You will not always be given a visible "Next" link. When a page has no navigation markup, hunt for the pattern in the links it does contain: ?page=, ?p=, ?start=, ?offset=, or ?per_page=. The moment you see an offset, you can drive the whole series without touching the UI. A disciplined loop also needs a stopping rule, and the most defensive one is a repeat guard: remember the first row of each page and stop if a page starts with a row you already saw. That catches infinite redirect loops and "page 9990 is just page 2 again" servers that never return an empty page.

Budget the Loop, Don't Trust It

Being generous with a while-loop is how a scraper fetches four hundred thousand pages by accident. Budget every pagination loop explicitly, and treat "hits the cap" as a signal to inspect, not as silent success:

def scrape_paginated(client, url, page_param="page", cap=500):
    for page in range(1, cap + 1):
        resp = client.get(url, params={page_param: page}, timeout=15.0)
        rows = extract_rows(resp.text)
        if not rows:
            break
        yield from rows

Pagination is the first loop in this course that can run away from you, and it will not be the last. The pattern you are learning here — enumerate, stop on empty, cap the extremes, guard against repeats — reappears verbatim in cursor loops, sitemap walks, and infinite scroll.