Capstone: design a URL shortener
Time to build something real and feel the whole toolkit click together. We're making a URL shortener: you paste in a long, ugly link and it hands back a short code like aB3xZ; later, anyone who visits short.ly/aB3xZ gets bounced straight to the original. That's it — two little operations. But to make it fast for millions of people, we'll reach for almost every piece you've learned. Let's walk it end to end.
Step 1 — What are we actually building? Two operations.
Strip it down and a URL shortener only ever does two things. One: SHORTEN. Someone gives us a long URL ('https://example.com/articles/2026/...') and we give back a short code, like aB3xZ. We remember 'aB3xZ means that long URL.' Two: REDIRECT. Someone visits short.ly/aB3xZ. We look up what aB3xZ means and send their browser to the long URL. Done. The huge insight that shapes everything else: redirects happen WAY more often than shortens. One link gets created once but might be clicked a million times. So the system's real job is doing redirects blazingly fast — keep that in your back pocket.
Step 2 — How do we make the short code?
We need a tiny, unique tag for each link — short so the URL stays small, and unique so no two links collide. The simplest trick: give every new link a plain counting number (1, 2, 3, ...) so it's automatically unique, then squeeze that number into a short code. To squeeze it, we use base62. That sounds scary but it's simple: instead of writing the number using only the 10 digits 0–9, we also allow the 26 lowercase letters and 26 uppercase letters — 62 characters in total. With 62 symbols per slot instead of 10, codes get short fast: just a handful of characters can name billions of links. That's why aB3xZ can stand for a link number in the billions while staying five characters long.
Step 3 — Where do we store the mapping? (→ sharding)
Every 'code → long URL' pair has to be saved somewhere we can look it up later: a database. For a few links, one database is plenty. But a popular shortener holds billions of mappings — far more than one machine can comfortably store and search. This is exactly the problem SHARDING solves: split the data across many databases, each holding a slice. We can shard by the code itself — codes starting one way live on shard 1, another way on shard 2, and so on. To find a mapping we jump straight to the shard that owns that code, so each database only ever holds and searches its own fraction. (See the sharding topic — this is its textbook use.)
Step 4 — Make redirects fast. (→ caching)
Remember Step 1: redirects vastly outnumber shortens, and a tiny handful of links (a viral tweet, a campaign link) get hammered over and over. Hitting the database for every single click of the same hot link is wasteful. So we put a CACHE in front of the database — a small, super-fast memory that remembers the answers to the most popular lookups. First click of aB3xZ: we ask the database, get the long URL, and tuck it in the cache. Every click after that: we find it in the cache and skip the database entirely. Since reads dominate and a few links are red-hot, the cache answers the overwhelming majority of redirects in a flash. (This is the read-heavy, hot-key case from the caching topic.)
Step 5 — The redirect flow, all together
Now watch a single click travel through the whole system. The browser asks for short.ly/aB3xZ. A load balancer points that request at one of our servers. The server checks the cache: if the link is hot, it's right there — redirect instantly, never touching the database. That's the common, fast path. Only on a cache miss (a cold link nobody's clicked lately) does the server go to the database — and sharding means it goes straight to the one shard that owns aB3xZ, not all of them. It gets the long URL, sends the redirect, and pops the answer into the cache so the next click is fast too.
Step 6 — Protect it. (→ rate limiting)
What stops someone from running a script that creates ten million junk links a second, or pounds our redirect endpoint to knock it over? RATE LIMITING. We put a limiter at the front door (right by the load balancer) that says 'you get this many requests per minute' per user. Spend your tokens too fast and you're politely told to wait. The flood never reaches our real servers, so one bad actor can't ruin the service for everyone else. (Straight from the rate limiting topic.)
Step 7 — Scale the servers. (→ stateless + load balancer)
Finally, the servers doing all this lookup-and-redirect work. We keep them STATELESS — no server stores anything special about you between requests; the real data lives in the shared database and cache. Because every server is interchangeable, we can run a whole fleet of them, and a LOAD BALANCER spreads incoming clicks evenly across the fleet. Need to handle more traffic? Add more identical servers. If one dies, the load balancer just stops sending it work and the others carry on. And that's the satisfying part: caching, sharding, rate limiting, stateless servers, load balancer — every tool you learned, each doing exactly one job, snapping together into one real system.
Questions you might have
▸What if two different long URLs accidentally get the same short code?
We design so that can't happen. Because each new link gets its own unique counting number (1, 2, 3, ... never repeated), and base62 turns each distinct number into a distinct code, two different links always get different codes. The uniqueness is baked in at the source — we never generate the same code twice.
▸Why not just store all the mappings in one big file?
A single file works for a toy, but a real shortener holds billions of mappings and gets thousands of lookups a second. One giant file on one machine is too big to fit comfortably and too slow to search, and if that one machine dies, everything is down. That's exactly why we use a database, split it across shards, and put a cache in front — to stay fast, fit the data, and survive a failure.
▸If two people click the same short link at the same instant, do they clash?
No — a redirect is a read. Both clicks just look up the same 'code → long URL' answer and both get sent to the same place. Reads don't interfere with each other, which is also why caching works so well here: the cache can hand the same answer to a thousand people at once without any trouble.
▸What happens the very first time a brand-new link is clicked — isn't the cache empty for it?
Right, that first click is a cache miss: the cache doesn't know the link yet, so the server fetches it from the (sharded) database, sends the redirect, and stores the answer in the cache on the way out. Every click after that finds it in the cache and is instant. The cache fills itself up with whatever turns out to be popular.
▸Why did we need ALL these pieces for something this simple?
Two operations are simple; doing them for millions of people, fast, without falling over, is the hard part — and that's what each tool handles. Sharding makes the mountain of data fit, caching makes the flood of reads fast, rate limiting keeps abusers out, and stateless servers behind a load balancer let us add machines and survive failures. The shortener is simple; making it big and reliable is the whole craft of system design.
Best read after: Caching, Sharding, Rate limiting