The Depth Behind the Selectors

The previous lessons used BeautifulSoup and lxml as if they were the same thing. They are not, and the difference matters the moment you parse more than a few thousand pages. Parsing is where scraped HTML becomes usable data, and the engine you choose sets your speed ceiling and your tolerance for broken markup. This lesson covers the real parser options, their trade-offs, and the encoding layer that silently ruins datasets nobody suspects.

The Parser Menu

  • BeautifulSoup + html.parser — the standard library bundle. Pure Python, forgiving, and the slowest. Fine for quick scripts and exploration.
  • BeautifulSoup + lxml — BeautifulSoup's API on lxml's C engine. The sweet spot for readable code at real speed. This is the default for production Python scrapers.
  • lxml directly — the fastest mainstream option, with first-class XPath from the previous lesson and almost no overhead.
  • selectolax — a newer, hyper-optimized C bindings parser. The fastest when raw throughput is the whole game, but its API is brighter and less forgiving with mangled HTML.

The rule of thumb: start with BeautifulSoup(html, "lxml"). Measure with the timing script below before moving to selectolax. Never run html.parser on a million pages by accident.

from lxml import html as lh
from bs4 import BeautifulSoup
from timeit import timeit

html_text = open("page.html", encoding="utf-8").read()

print("lxml        ", timeit(
    "lh.fromstring(html_text)", globals=globals(), number=20))
print("bs4 + lxml  ", timeit(
    'BeautifulSoup(html_text, "lxml")', globals=globals(), number=20))
print("html.parser ", timeit(
    'BeautifulSoup(html_text, "html.parser")', globals=globals(), number=20))

On a real page the gap between html.parser and lxml is routinely a step of many times. Choose the engine first, then write the code around it once.

Why Encoding Breaks Datasets

A page announces its encoding in the Content-Type header (charset=) and again inside a <meta charset> tag. When they disagree, or when a library guesses wrong, you get mojibake: the UTF-8 text "café" displayed as café, or a Russian title turned into wall art of replacement characters. Your scraper then stores garbage without any error, and the dataset is poisoned quietly. Encoding is the failure that never raises.

requests and httpx default to ISO-8859-1 when no charset is given, because that was the HTTP 1.0 habit. For most modern pages that is wrong.

resp = client.get(url, timeout=15.0)
if resp.encoding is None or resp.encoding.lower() in ("iso-8859-1", "latin-1"):
    resp.encoding = resp.apparent_encoding
text = resp.text

apparent_encoding heuristically detects the charset from the raw bytes using chardet-style analysis. Trust it only when the real charset is missing; a header that says utf-8 is usually right even when odd bytes appear. If you see � replacement characters or pairs like é in your output, do not change the extractor, change the encoding logic: parse again from raw bytes, not from an already-mangled string.

Practical Crossing

Correct order of operations when a page gives you trouble: fetch raw bytes, resolve the encoding (header, then meta, then heuristic), decode once, and only then hand the string to your parser. Libraries that decode silently too early (BeautifulSoup via resp.text) mean the encoding bug has already happened before your selectors run. When correctness matters, call client.get with content semantics and choose the encoding yourself:

raw = client.get(url, timeout=15.0).content
enc = "utf-8"
text = raw.decode(enc, errors="replace")

And when a site serves split payloads or ISO-page encodings you do not trust, keep a sample of every tenth response's first 4KB with its declared charset, the same habit the data-cleaning lesson teaches for layouts. Encoding rot, like layout drift, is far easier to fix when you have an artifact of the original to compare against.