~11 min
A message queue sits between a producer and a consumer and buys a specific kind of slack: the producer can keep accepting work even when the consumer is temporarily slow, down, or scaling up, because the work waits in the queue instead of piling up against a synchronous call that has to succeed immediately. That's the core trade a queue makes over a direct API call — a producer that calls a downstream service directly is coupled to that service's uptime and latency; a producer that enqueues a message is coupled only to the queue's uptime, which is usually the easier thing to make highly available. The cost is that the work is no longer synchronous: the producer doesn't get an answer back, only a confirmation that the message was accepted, so a queue fits work whose result the caller doesn't need immediately — sending an email, resizing an image, charging a card asynchronously — and fits poorly where the caller needs a same-request answer.
Once a message is enqueued, a consumer has to be able to pick it up without another consumer picking up the same one — and a queue can't just delete a message the moment it's handed out, because the consumer might crash before finishing. AWS's own documentation on SQS describes the mechanism it uses: receiving a message doesn't delete it, it makes the message invisible to other consumers for a visibility timeout, and the consumer is responsible for explicitly deleting the message once it's actually done. If the timeout expires before the message is deleted — because the consumer crashed, or simply took too long — the message becomes visible again and another consumer can pick it up. That single mechanism is also why SQS's documentation is explicit that visibility timeout is not a guarantee against a message being delivered twice.
That "not a guarantee against duplicate delivery" is the load-bearing fact of the whole pattern: a queue offering at-least-once delivery means a consumer has to be written to handle processing the same message twice without a bad outcome — charging a card once for a message that got processed twice, not twice. A consumer that isn't idempotent will eventually double-process something, because the timeout-and-redeliver mechanism that makes the queue resilient to a crashed consumer is the same mechanism that occasionally hands out a duplicate. A separate, related mechanism handles messages that keep failing rather than duplicating: after a message has been received and has failed to be deleted some configured number of times, it's routed to a dead-letter queue instead of being redelivered forever, so a single poison message doesn't block the whole queue behind it.
The visibility timeout expires and the message is redelivered to another consumer, which will attempt the same charge again — so the charge logic has to be idempotent (e.g. keyed on a request ID) to avoid billing the customer twice.