The Data That Hides Inside Files

Lists, search results, and catalogs live in HTML. But the final answer lives in a PDF. Regulatory filings, annual reports, tenders, court dockets, bank statements, and public procurements all funnel their real content into documents, because documents are what humans print. Scraping was never finished if the pipeline stops at the download link. This lesson takes the file from the streaming lesson and turns it back into rows.

Detect What You Actually Got

A URL ending in .pdf lies; so does a Content-Type header. Content negotiation and wrappers mean the honest check is the first bytes of the body, the magic number. %PDF for PDFs, PK for docx/xlsx/zip, and the byte 0x89 followed by PNG for images. Peek before committing to a parser:

def sniff(data):
    if data[:5] == b"%PDF-":
        return "pdf"
    if data[:2] == b"PK":
        return "office-zip"
    if data[1:4] == b"PNG":
        return "png"
    return "unknown"

PDFs With a Text Layer: pdfplumber

Most generated PDFs embed extractable text. pdfplumber walks pages, returns text, and — the part that makes it the workhorse — extracts tables with coordinates, preserving structure that a naive text dump destroys.

import pdfplumber

with pdfplumber.open("report.pdf") as pdf:
    for page in pdf.pages:
        text = page.extract_text()
        for table in page.extract_tables():
            for row in table:
                print(row)

Tables are where the value usually is: invoice lines, population counts, transaction lists. When cells merge or reflow, extract_tables still returns a rectangular grid with None for the gaps; clean those with the normalisation tools from the cleaning lesson.

Scanned PDFs Need OCR

If extract_text() returns empty on every page, the PDF is a sheaf of images — a scan. The text layer does not exist, so you read pixels instead. The standard stack: convert each page to an image (pdf2image wraps poppler), preprocess, and pass through Tesseract via pytesseract.

import pytesseract
from pdf2image import convert_from_path
from PIL import Image, ImageOps

pages = convert_from_path("scan.pdf", dpi=200)
for i, image in enumerate(pages):
    gray = ImageOps.grayscale(image)
    threshold = gray.point(lambda p: 255 if p > 150 else 0)
    text = pytesseract.image_to_string(threshold)
    print("page", i, "->", text[:200])

The preprocessing line is worth the words it costs: upscale to at least 200 dpi, grayscale, and threshold before OCR. Tesseract's accuracy on clean contrast is dramatically higher than on raw scans, and the speed penalty is negligible. Budget for OCR errors: OCR text is best-effort, so validate numbers (they carry the money) twice, and keep the source image path in your records so a bad read is re-cuttable.

Office Documents and Spreadsheets

The PK magic precedes the modern zip-based formats. python-docx reads paragraphs and tables out of .docx; openpyxl (or pandas) reads .xlsx and .xls; .rtf and old binary formats get migrated through a converter. The extraction pattern is constant: open with the right reader, iterate the document's logical units (paragraphs/cells), normalise, and emit rows. The only real difference from parsing HTML is the reader — everything downstream belongs to cleaning, validation, and storage.

Route, Don't Special-Case

Resist the urge to scatter document handling through your main crawl. Build one extract_document(path) that sniffs magic bytes, dispatches to pdfplumber / tesseract / docx / spreadsheet readers, and returns a uniform list of normalized rows. Then the rest of your pipeline has no idea a file was involved at all, and adding a matching format is one small branch instead of archaeology through a long-dead scraping script.