Queues, Streams and Rate Limiting

A message queue sits between a producer and a consumer and buys a specific kind of slack: the producer can keep accepting work even when the consumer is temporarily slow, down, or scaling up, because the work waits in the queue instead of piling up against a synchronous call that has to succeed immediately. That's the core trade a queue makes over a direct API call — a producer that calls a downstream service directly is coupled to that service's uptime and latency; a producer that enqueues a message is coupled only to the queue's uptime, which is usually the easier thing to make highly available. The cost is that the work is no longer synchronous: the producer doesn't get an answer back, only a confirmation that the message was accepted, so a queue fits work whose result the caller doesn't need immediately — sending an email, resizing an image, charging a card asynchronously — and fits poorly where the caller needs a same-request answer.

Once a message is enqueued, a consumer has to be able to pick it up without another consumer picking up the same one — and a queue can't just delete a message the moment it's handed out, because the consumer might crash before finishing. AWS's own documentation on SQS describes the mechanism it uses: receiving a message doesn't delete it, it makes the message invisible to other consumers for a visibility timeout, and the consumer is responsible for explicitly deleting the message once it's actually done. If the timeout expires before the message is deleted — because the consumer crashed, or simply took too long — the message becomes visible again and another consumer can pick it up. That single mechanism is also why SQS's documentation is explicit that visibility timeout is not a guarantee against a message being delivered twice.

That "not a guarantee against duplicate delivery" is the load-bearing fact of the whole pattern: a queue offering at-least-once delivery means a consumer has to be written to handle processing the same message twice without a bad outcome — charging a card once for a message that got processed twice, not twice. A consumer that isn't idempotent will eventually double-process something, because the timeout-and-redeliver mechanism that makes the queue resilient to a crashed consumer is the same mechanism that occasionally hands out a duplicate. A separate, related mechanism handles messages that keep failing rather than duplicating: after a message has been received and has failed to be deleted some configured number of times, it's routed to a dead-letter queue instead of being redelivered forever, so a single poison message doesn't block the whole queue behind it.

A queue and a stream solve related but different problems, and the difference is what happens to a message after it's read. In a queue, a message is meant to be consumed once and then gone — that's exactly what deleting it after processing means. A stream, the shape Kafka is built around, keeps every published record for a configurable retention window regardless of whether anyone has read it yet, and lets multiple independent consumers read the same records at their own pace, each tracking its own position. That difference matters for a specific class of design: an order-placed event that a fulfillment service, a notifications service, and an analytics pipeline all need to react to independently is a stream problem, not a queue problem — a queue message consumed by one of those services is simply gone for the other two.

Kafka's own documentation describes the structure directly: a topic is a category records are published to, and the cluster maintains it as a partitioned log — each partition an ordered, append-only sequence of records, where every record gets a sequential offset that identifies its position within that one partition. Splitting a topic into partitions is what makes a stream horizontally scalable in the same way sharding scales a database: each partition can live on a different broker and be consumed independently, so throughput grows with the partition count rather than staying capped at what one machine can push through.

Consumers read a topic as part of a consumer group, and Kafka's documentation is specific about the trade this makes: each partition is consumed by exactly one consumer within a given group at a time, which is what lets the group split the work of reading many partitions in parallel — but it also means there can never be more active consumers in a group than there are partitions, since the extras would have nothing to read. The ordering guarantee follows the same shape: Kafka guarantees order only within a single partition, not across the whole topic, so if two records need to be processed in the order they were produced, they need to land on the same partition — typically by keying them the same way sharding keys a table.

Rate limiting exists to protect a system from being overwhelmed — by a buggy client stuck in a retry loop, a traffic spike, or a client that's simply sending more than its fair share — by rejecting requests past a threshold instead of letting them degrade the service for everyone. AWS's own documentation frames this precisely: throttling settings are applied on a best-effort basis and should be thought of as targets, not guaranteed request ceilings, and a client that exceeds them receives a 429 Too Many Requests response rather than being served late or dropped silently. That distinction — reject fast with a clear signal, rather than queue everyone behind the overload — is what separates rate limiting from the buffering a queue provides; a queue absorbs a burst by making callers wait, a rate limiter protects capacity by turning some of them away.

The token bucket algorithm is the mechanism AWS's own documentation names for API Gateway's throttling: a bucket holds tokens, one token is spent per request, and tokens refill at a steady rate up to the bucket's maximum size. Two numbers describe the whole thing: the steady-state rate, how fast tokens refill (the sustained requests-per-second a client can send indefinitely), and the burst size, the bucket's maximum capacity (how many requests can arrive back-to-back, faster than the steady rate, before the bucket runs dry). A client sending well below the steady rate never empties the bucket and never gets throttled; a client sending a short burst above the rate gets to spend down the accumulated tokens first, and only starts seeing 429s once the bucket is empty and requests are still arriving faster than tokens refill.

Where the limit applies is its own design decision, separate from the algorithm. AWS documents throttling at several distinct scopes on the same service: an account-level limit shared across every API in a region, a per-API-stage limit set for one specific API, and a per-client limit tied to an individual API key — and these compose, with the tightest applicable limit winning. The same layering shows up in any rate-limited system: a global limit protects the whole service from an aggregate spike, while a per-client limit stops one noisy tenant from starving every other tenant of the shared capacity, and a design that only implements one of the two leaves the other failure mode completely unprotected.