Idempotency, Done Right

Idempotency means an operation applied twice has the effect of once. In payment systems it is the difference between “the webhook arrived twice” being a log line and being a double payout. The practice is straightforward — client-generated keys, server-enforced uniqueness, stored results — and the details that decide whether it actually works are the ones most guides skip.

The mechanics that matter

Keys are generated by the initiator. Whoever retries must send the same key on the retry. A server-generated key is worse than none, because it creates the appearance of protection while the second request gets a fresh key and proceeds happily. This has a corollary that trips teams up: the client must persist the key before the first attempt, so a client that crashes and restarts still retries with the original.

Uniqueness is enforced at the storage layer. A unique constraint on the key column, not an application-level check. Two requests racing past an if (!exists) is the textbook failure, and it happens precisely under the conditions that cause retries in the first place — load, timeouts, partial failures. Let the database resolve the race, because that is what databases are for.

The first result is stored and replayed. This is the part most implementations get wrong. When a duplicate arrives, returning an error is not idempotency — it breaks the caller who legitimately never received your first response. They asked twice because they do not know the answer; give them the answer. Store the response body and status against the key, and replay it.

Scope the key to the operation, not the endpoint. The same key arriving on “create payout” and “cancel payout” must not collide. Keying by (operation_type, key) avoids a class of bug that is extremely confusing to debug when it appears.

The decisions guides skip

Window

How long do keys live? Payments need days at minimum. Retries arrive from queues hours later; dead-letter reprocessing can be a day behind; and a gateway’s webhook redelivery policy may span longer than you expect — I have handled a Tuesday event redelivered on Friday, which a 24-hour window would have processed as new.

Pick a window that exceeds your longest realistic retry path, then partition the keys table by time and drop old partitions. Bounded growth, unbounded correctness within the window.

Storage

The keys table becomes one of your highest-write tables, and it grows monotonically until you manage it. Partition by creation time. Index only what you query — the key itself, scoped by operation. And resist the urge to store the full request payload for debugging convenience; store a hash for comparison and log the payload elsewhere, or the table becomes your largest object by an order of magnitude.

Failures, which are the actual hard part

A request that failed mid-flight with its key already recorded is the case that separates a working implementation from a plausible one. If the key exists but the operation never completed, replaying the stored result is wrong — there is no result. Re-executing blindly is also wrong if the effect partially landed.

So keys store state, not just existence: started, completed with a stored response, or failed. And the completion marker is written in the same transaction as the effect itself. That single discipline makes a half-finished request indistinguishable from one that never began, which is exactly what you need to be able to retry it safely.

Where the effect cannot be transactional with your own database — an external payout API, for instance — you need the external system’s own idempotency support, and you need to record the attempt before making the call. Most payment providers offer keys for exactly this reason; use theirs as well as yours.

Comparing request bodies

If the same key arrives with a different payload, something is badly wrong upstream — a client reusing keys, or a bug generating collisions. Silently replaying the first result hides it. Rejecting with a clear conflict error surfaces it while it is still cheap. Store a hash of the request and compare.

The fintech classic: the edge is not the ledger

Idempotency at your API edge does not absolve the posting layer. This is worth stating plainly because it is the most common misunderstanding I encounter in reviews.

The edge stops duplicate requests. It does nothing about a migration script that posts twice, an internal service that bypasses the API, a manual correction applied by an operator, or a replay of an event stream during a recovery. Every one of those reaches the ledger without passing your edge.

So the posting layer keeps its own invariants: per-event postings sum to zero, events carry their own natural uniqueness where one exists — a gateway’s transaction reference, for example, with a unique index on it — and reconciliation catches whatever slips through both. Defence in depth, because each layer’s failure mode is different. The ledger design covers the posting side; this article covers the edge; reconciliation is the net beneath both.

Retrofitting into a live system

Start with the write path where a duplicate costs the most, which is almost always payouts — because a duplicate payout has left your building and recovery depends on someone else’s goodwill. Then refunds, then payments.

The sequence that works without a freeze: add the key column with a unique constraint but do not require it yet. Start sending keys from your own clients. Monitor what percentage of traffic arrives with a key. When it is effectively all of it, make the key mandatory. The whole thing takes days rather than weeks, and it closes the largest ongoing source of ledger drift in most systems — which is why it sits second in the drift retrofit sequence, immediately after standing up reconciliation.

What good looks like

A system where the same key can be sent a hundred times and the effect happens once, the caller receives a consistent answer every time, a mismatched payload raises a conflict rather than hiding, and the keys table’s growth is bounded by a retention policy nobody has to remember.

None of this is difficult. All of it is easy to skip under deadline, and the cost of skipping arrives months later wearing a different name — usually “our numbers do not match” rather than “we did not implement idempotency.”

Testing it properly

Three tests that belong in your suite, because reasoning about this is not sufficient:

The duplicate test. Send the same key twice, assert the effect happened once and both responses are identical. Trivial, and it catches the replay-returns-error mistake.

The race test. Fire two identical requests concurrently and assert the same properties. This is the one that catches an application-level existence check masquerading as a constraint, and it fails on a surprising number of implementations that pass the sequential test.

The interrupted test. Kill the process midway through the operation, restart, retry with the same key, and assert exactly one effect. This is the state-versus-existence question made executable, and it is the test most implementations do not have — which is why the failure mode it covers is the one that shows up in production.

If those three pass, the mechanism works. If only the first passes, you have documentation rather than protection.

Related reading

/blog/ledger-design — the posting-layer invariants · /blog/why-your-ledger-drifts — what this prevents · /blog/event-driven-pitfalls — the same discipline in async systems · /services/fintech — having the write paths audited