Latency vs. throughput
When people say a system is "fast," they could mean two completely different things — and mixing them up leads to bad decisions. Latency is how long ONE request takes. Throughput is how MANY requests finish per second. They're not the same, and improving one can even hurt the other.
Picture a highway
Latency is about a single trip: how long does THIS car take to get from one end to the other? Measured in time — milliseconds. Throughput is about the crowd: how many cars make it across each minute? Measured in count-per-time — requests per second. A motorcycle has low latency (one rider zips across fast) but low throughput (only one rider). A wide highway packed with cars has high throughput (tons arrive per minute) even if each individual car is slow in traffic. See? Not the same number.
Why you can't just "make it fast" and call it done
Imagine your server answers one request in 100 milliseconds. That's its latency. But how many can it answer per second? If it handles them one at a time, that's only 10 per second — its throughput. A million people show up and most of them wait. So "fast for me" (latency) and "keeps up with everyone" (throughput) are different goals. Adding more servers usually raises throughput (more lanes) without changing latency (each car still crosses in the same time). And batching work can raise throughput while making each individual request WAIT longer — trading latency away to serve more.
Which one do you optimize?
It depends on what hurts. A typing-search box lives and dies by latency — every keystroke must feel instant. A nightly report that crunches a billion rows cares about throughput — nobody's watching, it just needs to finish the whole pile. The trap is optimizing the wrong one. Speeding up a single request won't save you if the problem is that ten million requests are queued behind it. Always ask: is my pain "each one is slow" or "there are too many"?
It's a tradeoff
| Option | 👍 Pro | 👎 Con |
|---|---|---|
| Optimize latency | Each request feels instant — great for interactive things (search, typing, games). | Doesn't help when the real problem is sheer volume; one fast request, a million still waiting. |
| Optimize throughput | Serves huge crowds; perfect for batch jobs and pipelines that just need to finish the pile. | Tricks like batching can make any single request WAIT longer — bad for interactive use. |
Questions you might have
▸Aren't "fast" and "handles a lot" the same thing?
No — that's the whole point. A motorcycle is fast (low latency) but carries one person (low throughput). A packed bus is slow per trip but moves lots of people (high throughput). Speed-for-one and volume-for-all are separate numbers.
▸What units are they measured in?
Latency is a time: milliseconds (ms). Throughput is a rate: requests per second (req/s). If you see a time, it's latency; if you see a 'per second', it's throughput.
▸Can I improve both at once?
Sometimes, but often there's a tug-of-war. Batching many requests together raises throughput but makes each one wait longer (worse latency). Adding more servers usually raises throughput without changing the latency of a single request. You have to know which one you're paying for.
▸Which one matters more?
Whichever one your users feel. Interactive apps (search, chat, games) live by latency. Background pipelines and reports live by throughput. Name your pain first, then optimize that one.
Best read after: What is a server?