Streams and Kafka

A queue and a stream solve related but different problems, and the difference is what happens to a message after it's read. In a queue, a message is meant to be consumed once and then gone — that's exactly what deleting it after processing means. A stream, the shape Kafka is built around, keeps every published record for a configurable retention window regardless of whether anyone has read it yet, and lets multiple independent consumers read the same records at their own pace, each tracking its own position. That difference matters for a specific class of design: an order-placed event that a fulfillment service, a notifications service, and an analytics pipeline all need to react to independently is a stream problem, not a queue problem — a queue message consumed by one of those services is simply gone for the other two.

Kafka's own documentation describes the structure directly: a topic is a category records are published to, and the cluster maintains it as a partitioned log — each partition an ordered, append-only sequence of records, where every record gets a sequential offset that identifies its position within that one partition. Splitting a topic into partitions is what makes a stream horizontally scalable in the same way sharding scales a database: each partition can live on a different broker and be consumed independently, so throughput grows with the partition count rather than staying capped at what one machine can push through.

Consumers read a topic as part of a consumer group, and Kafka's documentation is specific about the trade this makes: each partition is consumed by exactly one consumer within a given group at a time, which is what lets the group split the work of reading many partitions in parallel — but it also means there can never be more active consumers in a group than there are partitions, since the extras would have nothing to read. The ordering guarantee follows the same shape: Kafka guarantees order only within a single partition, not across the whole topic, so if two records need to be processed in the order they were produced, they need to land on the same partition — typically by keying them the same way sharding keys a table.