Message Queues vs Event Streams: Kafka Isn't Always the Answer
Both put a buffer between your services, so they get treated as interchangeable. They aren't. A queue hands each message to one worker and forgets it. A stream keeps an ordered log that many consumers read at their own pace. Choosing wrong means fighting your infrastructure for years.
Two Tools That Look Identical From the Outside
Somebody suggests decoupling two services with async messaging. Someone else says "let's use Kafka." Nobody asks whether the job needs a queue or a log, because from the outside they look the same: a producer writes, a consumer reads, and the two services no longer call each other directly.
Then six months in, the friction starts. You want to replay last Tuesday's events to backfill a new service and your queue deleted them the moment they were acknowledged. Or you're running a job queue on Kafka and discover you can't have 200 workers on a 12-partition topic, that a single poison message blocks its whole partition, and that "retry this one message in 30 seconds" is a genuinely hard problem.
Neither tool is broken. They're answers to different questions, and the difference comes down to one thing: what happens to a message after it's consumed.
The Core Distinction
A message queue is a work-distribution mechanism. A message goes to exactly one consumer in the group, and once acknowledged it is deleted. The broker tracks per-message state — delivered, acknowledged, retried, dead-lettered.
An event stream is an append-only log. Events are written in order, retained for a configured period regardless of who has read them, and each consumer independently tracks its own position (offset). Reading doesn't consume anything; ten consumers read the same event ten times, and an eleventh can start tomorrow and read it again.
Every other difference between RabbitMQ and Kafka descends from those two pictures.
What Follows From the Difference
| Message Queue (RabbitMQ, SQS) | Event Stream (Kafka, Pulsar, Kinesis) | |
|---|---|---|
| Message lifetime | Deleted on acknowledgement | Retained by time or size, independent of consumption |
| Consumers per message | One (per queue) | Unlimited, each with its own offset |
| Replay | No — it's gone | Yes — reset an offset and read it again |
| Ordering | Per queue, lost once workers run in parallel | Strict within a partition |
| Scaling consumers | Add workers freely; the broker load-balances | Capped by partition count |
| Per-message retry | Native — nack, requeue, delay, dead-letter | Manual — you build it, or block the partition |
| Routing | Rich — topics, headers, fanout, priority | Minimal — partition by key, filter in the consumer |
| Typical throughput | Tens of thousands/sec | Millions/sec |
| Mental model | A to-do list | A transaction log |
Three of these rows deserve more than a cell.
Consumer scaling is the one people hit first. In a queue, workers are interchangeable — 5 workers or 500, the broker just hands out messages. In Kafka, parallelism is bounded by partitions: a topic with 12 partitions supports at most 12 active consumers in a group, and the 13th sits idle. Partition count becomes a capacity decision you make in advance and can only increase (which reshuffles your key-to-partition mapping and breaks per-key ordering across the change).
Per-message retry is where queues quietly win. A job fails, you nack it, the broker redelivers it in 30 seconds, and after five attempts it goes to a dead-letter queue for a human to look at. All built in. In a log, there's no such thing as "retry this one message" — the consumer reads sequentially by offset. A message you can't process leaves you three bad options: skip it (data loss), block on it (the partition stops, and everything behind it waits), or write it to a separate retry topic and manage the redelivery yourself.
Ordering is the one people assume they have and don't. A queue with multiple workers gives you no ordering — message 1 goes to a worker that stalls, message 2 goes to a fast worker, and message 2 finishes first. A stream gives you strict ordering within a partition, which means ordering is guaranteed only among events that share a partition key. Key by user_id and every event for a given user is ordered; there is no global order across users, and there can't be, because that would mean one partition and no parallelism.
⚠️ Ordered and parallel are in direct tension
Any system that processes messages in parallel gives up total ordering — that's not an implementation weakness, it's arithmetic. What good systems offer is scoped ordering: strict within a key, unordered across keys. Design your consumers so that's enough. If you genuinely need a global total order, you're asking for a single-threaded consumer, and you should be sure the volume justifies the ceiling.
Choose By Naming the Thing You're Sending
The cleanest way to decide isn't to compare features. It's to look at what you're putting on the wire.
A command — 'do this thing'
ResizeImage, SendWelcomeEmail, GenerateInvoice. There is one correct handler, it should run exactly once, and when it's done the message has no further value. It has a natural retry policy and a natural failure destination. This is a job. Use a queue.
An event — 'this thing happened'
OrderPlaced, UserSignedUp, PriceChanged. It's a statement of fact about the past. The producer doesn't know or care who consumes it. Today it feeds the email service; next quarter it also feeds search indexing, fraud scoring, and a data warehouse — and those new consumers will want the history, not just what happens after they launch. Use a stream.
A record you'll want to re-derive state from
If a consumer's database is a projection of the event history — a search index, a read model, a materialized view — then the log is your source of truth for rebuilding it. When you fix a bug in the projection logic, you want to wipe the projection and replay from offset zero. Only a stream can do that. Use a stream, with retention long enough to rebuild.
✅ Fanout is the practical dividing line
Ask: is it plausible that a second, unrelated consumer will want these messages later? If yes, you want a stream, because retrofitting fanout onto a queue means either duplicating publishes to multiple queues (and updating the producer every time a consumer appears) or migrating. If a message is unambiguously a unit of work for one worker, a queue is simpler in every dimension and you should not reach for Kafka.
The Cost Nobody Puts in the Comparison Table
Kafka's throughput numbers are real, and they are almost never the constraint that matters for a team choosing between these. The constraint is operational.
A managed queue is close to zero-effort — SQS has no capacity to plan, no partitions to size, no consumer groups to rebalance. Kafka, self-hosted, is a distributed system you now operate: brokers, replication factors, ISR, partition rebalancing, consumer lag monitoring, retention and compaction policy, and a schema story so that a producer change doesn't break four downstream consumers at once. Managed Kafka removes some of that and none of the design decisions.
🔴 Pick the boring one until the log earns its keep
Most systems that adopt Kafka are using it as a queue with extra steps. That works, poorly, and the cost shows up as operational load and awkward retry code rather than as an outage. Start with a queue. Move to a stream when you have a concrete need — a second consumer of the same events, a replay requirement, event sourcing, or volume a queue genuinely can't hold. "We might need it later" is not that need; a queue can be migrated.
Worth knowing: the line is blurrier than it used to be. RabbitMQ streams add log semantics to a queue broker, Kafka has compaction and delayed-retry patterns, Redis Streams sits in the middle with consumer groups over a log, and NATS JetStream offers both. The decision framework holds even when one product can technically do both — you still have to decide which semantics you're relying on.
The Decision
Is the message a command to be executed once, or a fact that occurred?
Key Takeaways
- The difference is what happens after consumption. A queue deletes the message; a log retains it and lets each consumer track its own offset. Everything else follows from that.
- Queues distribute work; streams distribute facts. Commands (
SendEmail) belong in a queue. Events (OrderPlaced) belong in a log. - Replay is the stream's superpower, and it's the thing a queue can never give you. If you'll need to rebuild a projection or onboard a consumer that wants history, you need a log.
- Retry is the queue's superpower. Nack, delay, and dead-letter are native. In a log, a poison message either gets skipped, blocks its partition, or forces you to build a retry-topic pipeline.
- Partitions cap your consumer parallelism and define your ordering scope. Ordering exists only within a partition key — total ordering and parallelism are mutually exclusive.
- The real cost of Kafka is operational, not financial. Most teams reaching for it need a queue. Start simple; migrate when a second consumer or a replay requirement actually shows up.
References
- The Log: What every software engineer should know about real-time data's unifying abstraction — Jay Kreps — the argument the log abstraction is built on
- Kafka documentation — Design and consumer groups — partitions, offsets, and the limits of consumer parallelism
- RabbitMQ — Reliability and dead-lettering — acknowledgements, redelivery, and DLQ semantics
- Event-Driven Architecture: When Messages Solve Your Problems — the broader pattern these are the transport for