Load Balancing Strategies: Round Robin Is Just the Beginning
Round robin spreads requests evenly — right up until it hands the next request to a server that's already drowning. Even distribution of requests isn't the same as even distribution of load. Here's how the real algorithms decide where traffic goes, and when each one is the right call.
When "Even" Isn't Even
You put a load balancer in front of three servers and pick the obvious algorithm: round robin. Request 1 to server A, request 2 to B, request 3 to C, then back to A. Perfectly even. Every server gets exactly one-third of the requests.
Then one request to server A happens to be a giant report export that pins its CPU for 20 seconds. Round robin doesn't know or care — it keeps cheerfully sending A its fair share of new requests, piling them onto a server that's already underwater, while B and C sit half-idle.
That's the lesson that separates the naive view from the real one: an even number of requests is not an even amount of load. Requests have wildly different costs. The interesting load-balancing algorithms exist precisely because round robin is blind to that.
What a Load Balancer Actually Does
A load balancer sits between clients and a pool of backend servers, distributing incoming requests across them. It's the single entry point that lets you scale horizontally (add more servers) and stay available (route around failed ones). The interesting part is the policy it uses to choose a backend for each request — that policy is the strategy.
The Algorithms, and What Each One Knows
The strategies differ mainly in how much they know about the backends when they choose. More awareness generally means better distribution — at the cost of more state to track.
| Algorithm | How it chooses | Knows about load? | Watch out for |
|---|---|---|---|
| Round Robin | Next server in rotation | No | Slow requests pile onto busy servers |
| Weighted Round Robin | Rotation biased by server capacity | Static only | Weights are guesses; go stale |
| Least Connections | Server with fewest active connections | Yes — live | Long connections ≠ high CPU |
| Least Response Time | Fewest connections + lowest latency | Yes — live | Needs good latency measurement |
| IP / Hash | hash(client IP or key) → server | No | Uneven if keys skew; breaks on rescale |
| Power of Two Choices | Pick 2 at random, take the less-loaded | Yes — sampled | Simple + surprisingly effective |
A few worth understanding beyond the table:
Least Connections is the common upgrade from round robin. Instead of blindly rotating, it sends each new request to whichever server currently has the fewest open connections — a live proxy for "who's least busy." It naturally routes around that server stuck on a 20-second export.
Weighted variants let you handle a heterogeneous fleet — if server C has twice the cores, give it weight 2 and it receives twice the traffic. Useful, but the weights are static assumptions that drift as hardware and workloads change.
Power of two choices is the elegant one: pick two servers at random, send to whichever has fewer connections. It gets almost all the benefit of checking every server's load without the overhead of actually doing so, and it avoids the "everyone stampedes the single least-loaded server" problem that pure least-connections can cause at scale.
✅ Least Connections is a great default
For most web workloads with mixed request costs, least connections (or least response time) beats round robin without much thought. It reacts to actual load instead of assuming every request is equal. Reach for round robin only when requests really are uniform and cheap, or when you need dead-simple, stateless distribution.
The Session Problem: Why Hashing Exists
There's one thing the load-aware algorithms are bad at: sending the same client to the same server every time. Sometimes you need that — a server holding a user's in-memory session, or a cache warmed for a particular key.
⚠️ Sticky sessions are a trap in disguise
Pinning a user to one server (session affinity) means that server now holds state you can't lose. When it dies or you deploy, those users lose their session. It also undermines load balancing — you can't freely move a pinned user to a less-busy server. The better fix is usually to make servers stateless: push session state to a shared store (Redis) so any server can handle any request, and let the balancer optimize for load instead of stickiness.
When you genuinely need affinity, consistent hashing is the tool — it maps keys to servers such that adding or removing a server only remaps a small fraction of keys, instead of reshuffling everything (which plain hash % N does the moment N changes).
Layer 4 vs Layer 7
Load balancers operate at one of two levels, and the choice determines what they can see.
| Layer 4 (Transport) | Layer 7 (Application) | |
|---|---|---|
| Operates on | TCP/UDP connections, IPs, ports | HTTP — URLs, headers, cookies |
| Can route by content | No — it doesn't read the request | Yes — path, host, header-based routing |
| Speed | Faster, less overhead | Slower — parses the request |
| Use for | Raw throughput, non-HTTP traffic | Smart routing, /api vs /static, A/B, auth |
A Layer 4 balancer just forwards connections — fast and protocol-agnostic. A Layer 7 balancer reads the HTTP request, so it can route /api to one pool and /images to another, split traffic for canary releases, or terminate TLS. Most modern web setups use L7 for the intelligence; L4 shows up where raw speed or non-HTTP protocols matter.
Health Checks: The Part That Makes It "Highly Available"
Distributing load is only half the job. The other half is not sending traffic to dead servers. A load balancer continuously health-checks its backends and pulls unhealthy ones out of rotation automatically.
Do requests have wildly different costs (some cheap, some expensive)?
🔴 Check readiness, not just liveness
A naive health check (is the port open?) can keep sending traffic to a server that's up but broken — its database connection is dead, or it's still warming up. Check a real readiness endpoint that reflects whether the server can actually serve requests. And make the check frequent enough to eject a bad node fast, but not so aggressive that a brief blip evicts a healthy one.
Key Takeaways
- Even requests ≠ even load. Round robin distributes counts, not work, so it piles onto busy servers when request costs vary.
- Least Connections / Least Response Time react to live load and are the better default for mixed workloads. Weighted variants handle heterogeneous hardware. Power of two choices approximates load-awareness cheaply at scale.
- Sticky sessions trade load-balancing freedom and resilience for affinity — usually the wrong trade. Prefer stateless servers with a shared session store; use consistent hashing when affinity is truly required.
- Layer 7 balancers route on request content (paths, headers) and enable smart routing; Layer 4 is faster and protocol-agnostic.
- Load balancing without health checks isn't high availability. Check readiness, eject bad nodes fast, and don't evict healthy ones on a blip.
References
- The power of two random choices — Marc Brooker (AWS) — why sampling two beats picking the global minimum
- NGINX — HTTP load balancing methods — round robin, least conn, hash, and weighting in a real proxy
- Consistent Hashing — how it keeps keys stable across rescale — the mechanism behind affinity without full reshuffles