The Other Selection Language

CSS selectors find elements by their look. XPath finds them by their position in the tree and their content. When you need "the row that contains the word Total", "the second link in every cell", or "the parent of this button that also has a price", CSS shrugs and XPath answers. Real scrapers use both: CSS for fast, visual targets, XPath for anything relational or text-based. XPath is also what lxml, Selenium, and every XML/HTML tooling understands natively, so the skill transfers everywhere.

The Syntax in Fifteen Minutes

An XPath expression is a path from a location. // means "anywhere in the document", a single / means "directly below the current node", and .// means "anywhere below the current node". You navigate with element names, and you filter with square-bracket predicates.

  • //h3 — every <h3> anywhere on the page.
  • //div[contains(@class, "product")] — every div whose class contains the word "product" (the safe way to match multi-class elements).
  • //title/text() — the text content of the <title> tag.
  • //tr[position()=1] — the first table row; //tr[last()] — the last one.
  • //a[@href] — links that have an href attribute.
  • //td[normalize-space(.) = "Total"] — the cell whose text is exactly "Total".

Notice what XPath can do that CSS cannot: reach text, count positions, compare values, and walk upward to a parent or sideways to a sibling. The text() function returns the direct text of a node, and . refers to the current node's whole text.

Selection in lxml

lxml compiles XPath expressions into fast C code, so tree.xpath(...) is the natural home for this selector language.

from lxml import html as lh

tree = lh.fromstring(resp.text)

cards = tree.xpath('//div[contains(@class, "product-card")]')
for card in cards:
    title = card.xpath('.//h3/a/text()')
    price = card.xpath('.//span[contains(@class, "price")]/text()')
    print(title[0] if title else None, price[0] if price else None)

The leading . in .//h3 is what keeps each search inside one card instead of wandering off to the first match on the page. It is the XPath equivalent of scoping a CSS query to a subtree, and forgetting it is the single most common XPath bug.

The Workhorse Functions

Three functions handle nine out of ten real extraction problems.

  • contains(@class, "word") — partial attribute matching. Load-bearing because class attributes hold several classes and exact matches break.
  • normalize-space() — collapses whitespace and trims. Use it when the HTML is full of newlines inside the content you want.
  • text() vs . — text() is a node-list of the element's direct text; . evaluates to the full text including children. card.xpath('normalize-space(.)') is the "give me all the visible words in this element" incantation.
raw = tree.xpath('//div[contains(@class, "price-box")]')
for node in raw:
    print(node.xpath("normalize-space(.)"))

Positional and Structural Patterns

Tables are XPath's natural habitat. "The second cell of every row" is //tr/td[2] and "every total row" is //tr[contains(., "Subtotal")]. When a price sits in a sibling of a label, walk sideways:

price = tree.xpath(
    '//dt[normalize-space(text()) = "Price"]/following-sibling::dd[1]/text()'
)

following-sibling, preceding-sibling, parent::, and ancestor:: are the axes that let you extract from the shape of a page rather than from the immediate element. They are the difference between a scraper that survives a layout tweak and one that dies with it.

When to Reach for XPath Over CSS

Use XPath when any of these are true: you are targeting by visible text, by position, or by a parent/child relationship; the target's class list is unstable but a neighbor is stable; you are scraping tables; or you need normalize-space before you can trust a value. Use CSS when an id or a single class is stable, because it reads clearer and fails more obviously. Every DevTools panel offers Copy XPath, which is fine for a starting point, but rewrite those verbose //*[@id="..."] injections with a clean //div[@id="..."] expression you actually understand.