The Document Object Model (DOM)

Once you retrieve the raw text from an HTTP response, you are usually left with a massive string of HTML. To extract meaningful data, you must understand how HTML is structured into the Document Object Model (DOM).

HTML uses semantic tags (<div>, <span>, <h1>, <a>) organized in a hierarchical tree. Every element can contain attributes, text nodes, and child elements.

Identifying Data Targets

Before you write a single line of parsing code, you must inspect the target website: 1. Open your browser's Developer Tools (F12 or Right-Click -> Inspect). 2. Hover over the data you want to extract (e.g., a product price or article title). 3. Look at the surrounding HTML tags and their attributes (specifically class and id).

The Importance of Attributes

Classes and IDs are the hooks that CSS uses to style elements, and conveniently, they are the exact same hooks we use for web scraping.

  • IDs (id="product-price"): Should be unique per page. They are the easiest and most reliable way to target an element.
  • Classes (class="price-tag text-bold"): Often applied to multiple elements (like a list of products). You will frequently iterate over elements sharing a specific class.
  • Data Attributes (data-sku="12345"): Modern web apps frequently embed clean data directly into custom data-* attributes for JavaScript to use. These are goldmines for scrapers.
<!-- Example DOM Structure -->
<div class="product-card" data-sku="98765">
    <h2 class="product-title">Mechanical Keyboard</h2>
    <span class="product-price">$99.99</span>
    <a href="/products/mech-keyboard" class="btn-buy">Buy Now</a>
</div>

The Parsing Mindset

Your goal as a scraper is to define a path through this tree. For example, to get the price above, your mental model is: Find the div with class product-card, then find its child span with class product-price, and extract its inner text.

The Containers Data Actually Lives In

Most of the web's data sits in a handful of structural patterns you will learn to recognize on sight:

  • Tables (<table>, <tr>, <td>): specifications, price grids, and every financial figure ever published.
  • Definition lists (<dl>, <dt>, <dd>): the favorite of product pages, where <dt> is a label and <dd> its value ("Brand" -> "Acme").
  • Unordered/ordered lists (<ul>, <ol>, <li>): nav menus, tag feeds, comment chains.
  • Section wrapper divs (<div> with class="card", item, result): the repeating unit of catalogs and search results.

Every one of these is a "repeat a container, map through its children" structure, and recognizing the pattern in five seconds is what separates fast extraction from hunting.

Text Nodes vs. Elements

An element's visible text is rarely a single clean string. "Price: $99.99" often sits as several adjacent text nodes (from inline markup, icons, and whitespace). Pick the extraction tool that matches the structure: a CSS selector for the element, then its .text (BeautifulSoup) or text_content() (lxml) for the combined run of text inside it. Getting the "visible text" from an element with nested children is the step beginners skip, and it is the usual source of the mysterious [] list when extracting with XPath text().

Recon Your Page Before Writing Selectors

One script can do the discovery for you: parse the page, count the structural containers, and list the data-* attributes that modern frontends love to embed as ready-made extraction hooks.

from bs4 import BeautifulSoup
import re

soup = BeautifulSoup(html, "html.parser")
for table in soup.select("table"):
    print("table with", len(table.select("tr")), "rows")
print(sorted(set(re.findall(r'data-([a-z-]+)=', html)))[:20])

If the page exposes data-sku, data-id, or data-price attributes, you have just found a stable, machine-readable contract — often more stable than the visible markup around it.