The Web Is a Series of Forms

A browser does nothing but fill forms and follow the trail of responses. Login, search filters, checkout, "load more", and every wizard that a citizen portal or procurement system makes you click through: all of it is form submission wearing a costume. The scraping corollary is that a large class of targets is not crawled at all, it is submitted. Understanding forms and their state is how you get data out of them. Two skills cover most of it: serializing a form faithfully, and walking a multi-step flow with one persistent session.

Serialize the Form, Not the Guess

A form is a contract between the page and the server. It declares an action URL, a method (GET or POST), and a list of <input>, <select>, and <textarea> fields. Hidden inputs carry server-side state; do not invent values for them, read them from the page. The reliable way to submit a form is to parse the form and reuse what the page already populated.

from lxml import html as lh

tree = lh.fromstring(page_html)
form = tree.xpath('//form[@id="search-form"]')[0]

action = form.get("action")                 # destination path
method = (form.get("method") or "get").lower()

data = {}
for inp in form.xpath(".//input"):
    name = inp.get("name")
    if name:
        data[name] = inp.get("value", "")
for sel in form.xpath(".//select"):
    name = sel.get("name")
    if not name:
        continue
    chosen = sel.xpath('./option[@selected]/@value')
    data[name] = chosen[0] if chosen else (sel.xpath('./option/@value') or [""])[0]

if method == "post":
    resp = client.post(action, data=data)
else:
    resp = client.get(action, params=data)

Note what is missing: no guessing whether a field should be "checked" or how the site "probably" encodes things. data= gives you application/x-www-form-urlencoded; the browser's default for POST forms. If the form has enctype="multipart/form-data" (file uploads), switch to the files= parameter with the same field names.

Multi-Step Wizards Are a State Machine

A wizard spans several requests, and each step expects input minted by a previous step: a step token, a CSRF value, an item identifier, or a session cookie. The failure mode is submitting step two with step-one's token or a fresh client that never saw step one. The fix is discipline: one httpx.Client per wizard run (cookies persist), and each step's data parsed from the current step's response before the next request.

def step(html_text, step_name, extra=None):
    tree = lh.fromstring(html_text)
    form = tree.xpath('//form[contains(@data-step, "{}")]'.format(step_name))[0]
    data = {i.get("name"): i.get("value", "") for i in form.xpath(".//input") if i.get("name")}
    if extra:
        data.update(extra)
    return client.post(form.get("action"), data=data)

page1 = client.get("/apply").text
page2 = step(page1, "1", {"deg": "bsc"}).text
page3 = step(page2, "2").text

The one-session rule matters more than the tokens: if the server keeps wizard state in your session cookie, any new client restarts the flow and your step-two request starts answering for a wizard it never entered.

Files and Binary Payloads

Uploading a file through a multipart form belongs to the same story. The browser builds a boundary-delimited body; httpx does it for you from a files dict.

files = {"avatar": ("me.png", open("me.png", "rb"), "image/png")}
resp = client.post("/profile", data={"bio": "scraper"}, files=files)

For docs and attachments, stream from disk instead of loading the file into memory first, and always close the handle. Multipart bodies are large and the request timeout should reflect the real transfer.

Reading the Response, Not the Status Code

Forms fail silently: the server may return 200 with an error block "email already registered". Do not gate success on the status code alone. Parse the response for the fields you expect, search for known warning markers, and only then consider the step committed. The same discipline that validates scraped records applies to form submissions, because every wizard step is a network request producing records of its own.