Event-Driven Pitfalls

Ordering, replay and poison messages are the tuition fees of event-driven architecture, and everyone pays at least one. The pitfalls are predictable, which means the defences are designable — and the most useful defence is often deciding not to go event-driven for that particular flow at all.

First, the unpopular question: do you need this?

Event-driven architecture buys you three things: decoupling between producers and consumers, natural fan-out to multiple readers, and replay — the ability to rebuild state from history. Those are genuinely valuable, and they cost you ordering guarantees, debuggability and operational surface.

Most systems need that trade for one or two flows. The money path, where replay is what makes the ledger auditable. Anything fanning out to several independent consumers. Integration boundaries where the consumer’s availability should not affect the producer’s.

Most systems do not need it for everything, and adopting it everywhere converts every simple request-response interaction into a distributed debugging exercise. When I run an architecture review, the recommendation is more often “fewer topics than you planned” than “more” — because complexity is a budget, and the money path deserves most of it.

With that said: where you do need it, here is what will bite.

The big four

1. Ordering

Queues promise ordering per-partition at best, and consumers written on the assumption of global order eventually meet the refund processed before its payment.

The design that works: partition by entity key so all events for account 482 land on one partition and are therefore ordered relative to each other. Then, separately, make consumers tolerate out-of-order arrival anyway — because a redelivery, a rebalance or a replay will eventually deliver something out of sequence regardless of your partitioning.

Tolerance usually means the consumer checks state rather than assuming it: “apply this refund if the payment exists and is not already refunded” rather than “apply this refund.” That is slightly more code and it removes an entire category of 3 AM incident.

2. Duplicates

At-least-once delivery means at least. The same event arriving twice is not an anomaly, it is Tuesday — and the answer is the same discipline as the synchronous case, wearing asynchronous clothes: consumers are idempotent, keyed on the event’s own identity, enforced at the storage layer.

This is also the honest answer to anyone asking for exactly-once delivery. You cannot have it across a network. You can have at-least-once delivery plus idempotent consumers, which produces exactly-once effects — and that is what the question was really about.

3. Poison messages

One malformed event that crashes its consumer will crash it forever, blocking every message behind it. Without a dead-letter mechanism, a single bad payload can halt a queue indefinitely while the metrics show a healthy consumer that is simply not progressing.

Dead-letter after a bounded number of attempts, alert loudly, and give the DLQ a dashboard humans actually look at. That last part is the one that gets skipped: a silent DLQ is where financial events go to die quietly, and I have done the archaeology on more than one of them — a queue with months of accumulated failures, each one a real business event that never happened.

Make the alert actionable: event type, entity, consumer, error. A notification that only reports a count will be muted within a fortnight.

4. Replay

Replay is the superpower that justifies the whole architecture — rebuild any projection from history, recover from a bug by fixing code rather than migrating data. But it only works if it is rehearsed, and rehearsal requires two things.

Events must be facts, not commands. PaymentReceived is a fact; replaying it recomputes a projection harmlessly. SendReceipt is an instruction; replaying it emails your customer twice. This naming convention is not stylistic — it is a claim about replay safety, and a codebase that mixes the two has no safe replay path.

Side effects must be separable. Consumers that both update state and call external services cannot be replayed without either flagging the external calls off or pointing them at doubles. Design that seam deliberately, and practise using it in staging at real volumes. Replay you have never performed is an aspiration, and the first time you need it will be during an incident.

The subtler traps

Schema evolution. Events are immutable, which means an event written eighteen months ago must still be readable by today’s consumer. Version from the beginning, add fields rather than repurposing them, and never change the meaning of an existing field — because you cannot migrate history that is, by design, immutable.

The distributed transaction that is not. Publishing an event and updating your database are two operations, and if they are not atomic you will eventually have one without the other. The usual fix is an outbox: write the event to your own database in the same transaction as the state change, then publish from there. It is unglamorous and it removes the most common source of phantom or missing events.

Consumer lag as a business metric. Lag is not an infrastructure statistic — if the consumer that updates balances is twenty minutes behind, your balances are twenty minutes stale, and someone will make a decision on them. Surface lag where business people can see it, with a threshold that means something.

What the teardowns show

Across the fourteen companies I have taken apart publicly, the pattern is consistent: event-driven where volume and fan-out justify it, boring synchronous code everywhere else, and the money path treated with the most care regardless of which style it uses. Kaspi runs an enormous super-app with a monolith at its heart. Others went event-heavy and paid for it in operational complexity they now manage carefully.

Nobody who succeeded did it uniformly. That is the actual lesson, and it is the opposite of how the pattern is usually marketed.

An operational checklist

Six things to have in place before a topic carries anything financial:

Idempotent consumers, keyed on event identity, enforced by a constraint rather than a check.

Partitioning by entity key, so per-entity ordering is guaranteed even though global ordering is not.

A dead-letter queue with an actionable alert — type, entity, consumer, error — and a human who owns it.

An outbox for publishing, so an event and the state change it describes cannot diverge.

Consumer lag exposed as a business metric, with a threshold that reflects how stale the derived data may acceptably be.

A rehearsed replay, performed in staging at production volume, with side effects flagged off.

None of these are exotic. All of them are commonly missing, and each absence produces a failure that looks mysterious in production and obvious in hindsight.

Related reading

/blog/idempotency-done-right — idempotent consumers in detail · /blog/ledger-design — events as ledger facts · /blog/zero-downtime-migration — migrating event schemas · /services/architecture — deciding where this belongs