Scaling the Web

When a web application becomes too popular, a single server (no matter how powerful) will eventually buckle under the CPU and memory load. The solution is Horizontal Scaling: adding more servers.

But if you have 5 web servers, which IP address does the DNS return to the user? The answer is a Load Balancer. The Load Balancer acts as a reverse proxy. DNS points the user to the Load Balancer's IP. The Load Balancer accepts the incoming traffic and distributes it across the pool of backend servers.

Types of Load Balancers

1. Layer 4 Load Balancing (Network/Transport) Operates at the TCP/UDP level. It makes routing decisions based only on the Source/Destination IP and Port. - Pros: Blisteringly fast. Very low CPU overhead. - Cons: "Dumb". It cannot look inside the HTTP request. It doesn't know if the user is requesting /images or /api. - Use Case: High-throughput database traffic, raw TCP streams.

2. Layer 7 Load Balancing (Application) Operates at the HTTP level. It terminates the TCP connection, decrypts the TLS/SSL, reads the actual HTTP headers and URL path, and then opens a new connection to a backend server based on that data. - Pros: Highly intelligent. It can route traffic to /images to an optimized static file server pool, and /api to a heavy CPU server pool. - Cons: Slower, requires more CPU overhead for TLS termination. - Use Case: Web traffic, API gateways, microservice routing.

Load Balancing Algorithms

How does the balancer choose which server gets the next request? - Round Robin: Sequential allocation. Server 1, Server 2, Server 3, Server 1... (Good if all servers are identical). - Weighted Round Robin: Servers get more or fewer turns proportional to their capacity. Essential when your fleet mixes instance sizes or generations. - Least Connections: Sends the request to the server with the fewest active TCP connections. (Best for long-lived requests like WebSockets). - Least Response Time: Combines connection count with measured latency, routing to whichever healthy server is answering fastest right now. - IP Hash: Hashes the client's IP address. This guarantees that User A always connects to Server 2. This is called Sticky Sessions (necessary if your app stores session data in local server RAM instead of a centralized Redis cache).

Health Checks

A load balancer is useless if it routes traffic to a dead server. Balancers constantly ping backend servers (e.g., an HTTP GET to /health). If a server fails the health check 3 times in a row, the balancer removes it from the pool until it recovers.

Design meaningful health checks: a check that only verifies the process is running will happily route traffic to a server with a broken database connection. A deep health check verifies critical dependencies (database, cache, disk space) so unhealthy instances are drained before users notice.

Connection Draining

When you deploy a new version, you remove old instances from the pool—but in-flight requests may still be using them. Connection draining (or deregistration delay) keeps the old instance serving existing connections for a grace period while the balancer stops sending it new traffic. Without it, every deploy produces a burst of dropped requests.

Global Server Load Balancing

A single load balancer is still a single point of failure and a single geographic location. GSLB extends balancing across regions, using DNS or Anycast to steer users to the nearest healthy data center, with failover to another region if an entire site goes down. This is the layer that gives large services their resilience to regional outages.