What a Proxy Actually Changes

A proxy changes exactly one thing: the source IP address the server sees. It does not change your TLS fingerprint, your header order, your cookies, your timing, or your JavaScript. When a scraped site blocks you, assume the cause is your fingerprint or your behavior before you assume it is your IP, because paying a proxy for a problem that is not about your IP simply moves traffic to a different address that gets blocked the same way.

Why use one at all, then? When a server rate-limits by IP, a rotation gives you extra independent buckets to spread requests across. That is the only real benefit. If the limit is per account, per cookie, or per fingerprint, rotation does nothing for you.

Datacenter, Residential, Mobile

Read the listings carefully because the names mean real things:

  • Datacenter proxies sit in cloud providers that are known to be hosting machines. They are cheap, fast, and widely flagged. Fine for non-JS, non-competitive data, wasting money if the target runs a commercial anti-bot.
  • Residential proxies are IPs assigned to ordinary households and routed through broadband ISPs. The server sees a normal person's IP, which is why they cost an order of magnitude more. Use them only where a datacenter IP is provably being rejected and the data justifies the price.
  • Mobile proxies come from cellular carriers. They are rare and expensive, and are usually overkill unless the target serves a mobile-only experience.

Start with datacenter proxies on a single target, measure the block rate, and move up only when a real block rate shows up.

Keep Sessions Sticky

The classic novice mistake is rotating on every single request. Most targets store state in your session or account, and the moment the source IP changes, that state is orphaned. A login session is tied to the address that authenticated. Rotate, and the server sees a brand new visitor without credentials.

Assignment policy that works: pick one proxy per target host, use it for the entire crawl of that site, and only switch when that proxy starts failing. Pin the proxy to the host so a multi-site crawl does not shuffle addresses between targets.

import httpx

POOL = [
    "http://user:pass@dc-proxy-01.example.com:8080",
    "http://user:pass@dc-proxy-02.example.com:8080",
    "http://user:pass@dc-proxy-03.example.com:8080",
]
_next = 0
host_proxy = {}

def proxy_for(url: str) -> str:
    global _next
    host = httpx.URL(url).host
    if host not in host_proxy:
        host_proxy[host] = POOL[_next % len(POOL)]
        _next += 1
    return host_proxy[host]

def fetch(url: str, client: httpx.Client) -> httpx.Response:
    return client.get(url, proxy=proxy_for(url), timeout=15.0)

Retire Dead Addresses

Proxies die, and a pool with a dead entry burns a full timeout on every request it draws. Track consecutive failures per proxy and retire an address after a small threshold, then cycle it back in later. A simple counter dict is enough at this scale.

failures = {}

def guarded_fetch(url: str, client: httpx.Client):
    host = httpx.URL(url).host

    def pick():
        if host not in host_proxy or host_proxy[host] is None:
            host_proxy[host] = POOL[len(failures) % len(POOL)]
        return host_proxy[host]

    proxy = pick()
    for attempt in range(2):
        try:
            resp = client.get(url, proxy=proxy, timeout=15.0)
            if resp.status_code in (407, 429):
                raise httpx.TransportError("proxy refused " + str(resp.status_code))
            return resp
        except (httpx.ProxyError, httpx.TransportError):
            failures[proxy] = failures.get(proxy, 0) + 1
            if failures[proxy] >= 3:
                host_proxy[host] = None
                failures[proxy] = 0
                proxy = pick()
    return client.get(url, proxy=proxy, timeout=15.0)

Rotate against Pushback, Not the Clock

Do not tie rotation to a timer. Tie it to evidence. Rotate when the server says too many requests (pull a fresh IP and continue), when an address is dead, or when a consistent block pattern points at the address. Rotating on a schedule multiplies your proxy bill and makes sessions more fragile, and it does not help if the blocker is fingerprint-based.

The one rule worth treating as law: when you rotate, reuse the new address for everything from that host until it fails. Stability beats breadth on almost every target.