~12 min
Rate limiting exists to protect a system from being overwhelmed — by a buggy client stuck in a retry loop, a traffic spike, or a client that's simply sending more than its fair share — by rejecting requests past a threshold instead of letting them degrade the service for everyone. AWS's own documentation frames this precisely: throttling settings are applied on a best-effort basis and should be thought of as targets, not guaranteed request ceilings, and a client that exceeds them receives a 429 Too Many Requests response rather than being served late or dropped silently. That distinction — reject fast with a clear signal, rather than queue everyone behind the overload — is what separates rate limiting from the buffering a queue provides; a queue absorbs a burst by making callers wait, a rate limiter protects capacity by turning some of them away.
The token bucket algorithm is the mechanism AWS's own documentation names for API Gateway's throttling: a bucket holds tokens, one token is spent per request, and tokens refill at a steady rate up to the bucket's maximum size. Two numbers describe the whole thing: the steady-state rate, how fast tokens refill (the sustained requests-per-second a client can send indefinitely), and the burst size, the bucket's maximum capacity (how many requests can arrive back-to-back, faster than the steady rate, before the bucket runs dry). A client sending well below the steady rate never empties the bucket and never gets throttled; a client sending a short burst above the rate gets to spend down the accumulated tokens first, and only starts seeing 429s once the bucket is empty and requests are still arriving faster than tokens refill.
Where the limit applies is its own design decision, separate from the algorithm. AWS documents throttling at several distinct scopes on the same service: an account-level limit shared across every API in a region, a per-API-stage limit set for one specific API, and a per-client limit tied to an individual API key — and these compose, with the tightest applicable limit winning. The same layering shows up in any rate-limited system: a global limit protects the whole service from an aggregate spike, while a per-client limit stops one noisy tenant from starving every other tenant of the shared capacity, and a design that only implements one of the two leaves the other failure mode completely unprotected.
The first 40 succeed by spending the full bucket; the remaining 10 are throttled with 429s, since the bucket empties faster than it refills at only 10 tokens/sec.