Designing a Rate Limiter

The messaging module covered the token bucket algorithm conceptually — a rate and a burst size, tokens spent and refilled. Building one is a storage problem before it's an algorithm problem: if each of a fleet of stateless application servers keeps its own in-memory count of a client's requests, a client hitting a load balancer that spreads traffic across ten servers effectively gets ten times its intended limit, because no server knows what the other nine have counted. A rate limiter enforcing, say, 100 requests/minute per API key across a fleet of stateless servers needs its counters to live somewhere every server can see — which is precisely the shared-state problem a cache like Redis, already covered in this course, is built to solve.

A simple fixed-window version of this is a direct application of two commands this course already covered: INCR on a key like ratelimit:{api_key}:{current_minute} bumps the count, and EXPIRE on that same key, set once when the counter is first created, clears it automatically at the window boundary — the exact TTL mechanism from the caching module, repurposed as the reset clock for a rate-limit window instead of a cache-eviction clock. If the incremented count comes back over the limit, the request is rejected with a 429, the same response this course's rate-limiting lesson named for API Gateway's own throttling. The one new subtlety: checking the count and deciding whether to reject has to happen as a single atomic step against the store, not as a separate read-then-write from the application, or two concurrent requests from the same client can both read the count just under the limit and both be allowed through — the same class of race a lock or an atomic increment exists to close.

A fixed window has a real edge case worth naming: a client that sends its whole limit in the last second of one window and its whole limit again in the first second of the next window bursts to twice the intended rate across that boundary, even though neither individual window was exceeded. A sliding-window or true token-bucket-in-Redis implementation (tracking a weighted count across the previous and current window, or storing token count and last-refill timestamp directly) closes that gap at the cost of a more complex update than a single INCR. Which version is worth building is, like everything else in this course, a question with a number attached: a fixed window is enough when a client bursting to 2x for a few hundred milliseconds at a boundary is an acceptable risk; it isn't when that burst is exactly the failure mode the rate limiter exists to prevent.