Respecting the Target Server

Web scraping exists in a legally and ethically gray area. While public data is generally free to consume, the infrastructure serving that data costs money. A poorly written scraper can easily bring down a small server, essentially performing an unintentional Denial of Service (DDoS) attack.

Professional data engineers adhere to strict ethical guidelines to ensure the longevity of their scrapers and the health of the target servers.

The robots.txt Standard

robots.txt is a text file located at the root of a domain (e.g., https://example.com/robots.txt). It is a voluntary protocol where site administrators specify which parts of their site should not be crawled by automated bots.

User-agent: *
Disallow: /admin/
Disallow: /api/private/
Crawl-delay: 5

Before scraping a new target, you should inspect this file. If a path is disallowed, you should generally respect that boundary unless you have a specific, legally vetted reason to bypass it. The Crawl-delay directive indicates how many seconds you should wait between requests.

Core Ethical Guidelines

  1. Identify Yourself: Do not use fake user-agents unless you are actively trying to bypass malicious anti-bot protection on public data. Use a descriptive user-agent with contact information: User-Agent: MyDataBot/1.0 (bot@mycompany.com). If you break something, administrators can contact you instead of just blocking your subnet.
  2. Rate Limiting is Mandatory: Never unleash unbounded concurrent requests against a target. Throttle your spiders. 1-2 requests per second is a polite baseline for most servers.
  3. Cache Your Responses: During development, you will run your parsing logic hundreds of times. Do not fetch the live HTML every time you debug a CSS selector. Save the HTML payload to your local disk and parse the local file until your script is perfect.
  4. Scrape During Off-Peak Hours: If extracting massive historical datasets, run your jobs between 2 AM and 6 AM relative to the target's primary user base timezone.
  5. Only Extract What You Need: Do not download massive high-resolution images or videos if you only need the metadata. Optimize your requests to save bandwidth for both you and the server.

robots.txt Beyond the Basics

The User-agent: lines are not a single * rule; real robots.txt files define per-bot policies, and you should honor your own bot's section before panicking about the general one. Python's urllib.robotparser does this scoping for you:

import urllib.robotparser

rp = urllib.robotparser.RobotFileParser()
rp.set_url("https://example.com/robots.txt")
rp.read()

print("can fetch /products?", rp.can_fetch("*", "https://example.com/products"))
print("delay requested:", rp.crawl_delay("*"))

Two reading comprehension notes: Disallow: / blocks everything, Disallow: on its own blocks nothing, and an empty robots.txt sends a 404 that some libraries treat as permission and some as an error — decide once in your code. Crawl-delay is a hint honored by polite crawlers and ignored by others, which is why the rate-limiting lesson makes your own throttle the real authority.

The Unwritten Courtesy Layer

robots.txt is the floor, not the ceiling. The courtesy rules that keep you welcome on a site for years: cache aggressively in development so your debugging never touches their server (from the caching lesson), spread large crawls across a schedule instead of one burst, and keep a real contact address in your User-Agent. Most site operators block scripts they cannot identify and tolerate ones they can. Being reachable is not weakness, it is the difference between a warning email and a permanent IP block.