Load balancers
Once you have more than one server, a brand-new question appears: when a request shows up, WHICH server should answer it? If everyone always picked Server 1, the other servers would sit bored while Server 1 melted. A load balancer is the thing that decides — it spreads the requests across all your servers so no single one gets buried.
The restaurant host who seats you
Imagine a busy restaurant with one host at the door and ten waiters inside. You don't walk in and pick a waiter yourself — the host sends you to one. A good host watches who's free and spreads the tables out, so no single waiter is drowning while another stands around doing nothing. A load balancer is that host for your servers. Every request from the outside world arrives at the load balancer first. The load balancer doesn't do the real work itself — it just decides which server gets this request, then passes it along. To the user it's invisible: they think they're talking to one website, but behind the door there are ten waiters and a host quietly directing traffic.
How does it decide? Round-robin: just take turns
The simplest rule is the fairest one a kid would invent: take turns. Request 1 goes to Server A, request 2 to Server B, request 3 to Server C, then back to A for request 4, and so on around the circle. That's called round-robin — like dealing cards one at a time to each player around a table. It's dead simple and it works surprisingly well, because over thousands of requests the turns even out and every server ends up with about the same amount of work. There are smarter rules too — for example, instead of blindly taking turns, the load balancer can look at which server currently has the FEWEST jobs in progress and send the next request there (that's called least-connections). Round-robin is fair-by-counting; least-connections is fair-by-checking-who's-actually-busy.
Why this only works if the servers are stateless
Here's the catch that makes the whole trick possible. The load balancer might send your first request to Server A and your very next request to Server C. For that to be okay, ANY server has to be able to answer ANY request — it can't matter which one you land on. That's exactly what 'stateless' means: a server doesn't secretly remember 'oh, this is the user who was just here.' All the memory (who you are, what's in your cart) lives somewhere shared that every server can read, like a database. If servers were stateful — if Server A kept your shopping cart only in its own head — then bouncing you to Server C would lose your cart, because Server C never heard of you. So load balancing and stateless servers are a package deal: the load balancer is free to fling requests anywhere precisely because every server is interchangeable.
Bonus: it also notices when a server dies
A load balancer keeps a quiet eye on each server with little 'are you still alive?' check-ins (called health checks). If Server B stops answering — maybe it crashed — the load balancer simply stops sending requests to B and keeps dealing them to A and C. Users never notice; they just keep getting answers. This is the deeper reason load balancers matter. They don't only spread work; they let the system survive a server falling over. That's the resilience that horizontal scaling promised, finally delivered.
It's a tradeoff
| Option | 👍 Pro | 👎 Con |
|---|---|---|
| Round-robin (take turns) | Stupidly simple and fair: just deal each request to the next server in the circle. No measuring needed. | Blind — it doesn't notice if one request is a giant job. A server can get unlucky and pile up slow jobs while others coast. |
| Least-connections (send to the least-busy) | Smarter: it checks who currently has the fewest jobs and sends there, so a server bogged down with slow work gets skipped. | More work for the load balancer — it has to keep count of every server's live jobs, not just spin a wheel. |
Questions you might have
▸Wait, is the load balancer just another server?
Yes — it's a computer running a special program, but its program doesn't do your app's real work. Its whole job is to receive requests and hand each one to one of the real servers behind it. Think of it as the doorman, not one of the waiters.
▸If everything goes through the load balancer, isn't IT the weak point now? If it dies, everything dies.
Great catch — that's a real worry, and the answer is: in serious systems you run more than one load balancer, so if one fails another takes over. So no single load balancer is the lone point of failure. (You don't need the details now; just know engineers don't leave one lonely doorman in charge.)
▸Why can't each user just always talk to the same server? Wouldn't that be simpler?
You can pin a user to one server, but then you lose the two best things load balancing gives you. If that server dies, that user is stuck. And if lots of users happen to get pinned to the same server, it gets overloaded while others sit idle. Spreading requests freely across interchangeable servers avoids both problems.
▸What does 'stateless' have to do with load balancing?
Everything. Because the load balancer might send your next request to a different server, every server has to be able to answer it the same way — none of them can be secretly holding YOUR data in its own memory. Shared data (like a cart) lives in a database all servers read. That interchangeability is what lets the load balancer fling requests anywhere.
▸Is round-robin really good enough? It sounds too simple.
For a lot of real systems, yes! When requests are roughly the same size, taking turns evens out beautifully over thousands of requests. You only reach for smarter rules like least-connections when some requests are much heavier than others and blind turn-taking lets a server get unlucky.
Best read after: Stateless vs. stateful