Reconciliation at Scale

Reconciliation is proving your records against external truth — gateway settlement files, bank statements, custodian reports, physical counts. At small volume it is an Excel job. At scale it becomes an engineering discipline with a specific architecture: normalised feeds, a matching cascade, exception queues with reason codes, and metrics that tell you whether the upstream is rotting.

The pipeline

Ingest and normalise

Every external source speaks its own dialect. Each gateway’s CSV is a unique snowflake with its own column names, date formats and sign conventions; the bank’s statement format predates several of the countries using it; custodian files arrive as fixed-width text because 1987 never ended.

The first job is to normalise all of them into one comparison plane with five fields that matter: reference, amount, currency, business date, source. Everything else is kept but not compared. This step is unglamorous, it is most of the code, and doing it properly is what makes adding your fourth gateway a day’s work rather than a project.

The matching cascade

Exact match on reference plus amount, auto-cleared. This should clear the overwhelming majority — 95 per cent or better. If it does not, stop building the matcher and fix your references upstream, because a low exact-match rate means your records and theirs do not share a reliable identifier, and every clever technique downstream is compensating for that.

Tolerant match — amount within a defined rounding tolerance, or date within a settlement window — flags for review rather than clearing silently. The tolerance is logged with the match so it is auditable later.

Unmatched goes to the queue with a reason guess, and this is where reconciliation systems earn their keep. “Unmatched” is a shrug. “Gateway settled net of a refund we booked gross” is a fix. The taxonomy typically includes: missing counterpart, duplicate on either side, amount split across multiple settlements, timing across a cycle boundary, and fee treated differently by each side.

Building that taxonomy with your finance team is the highest-value conversation in the whole project. They already know the categories — they have been resolving them by hand for years — and encoding their knowledge is what converts an unmatched pile into a work queue.

Age and escalate

Items unresolved past thresholds climb: day three to the operations team, day seven to the finance lead, day fourteen to whoever owns the number publicly. Without escalation, the queue becomes a graveyard, and graveyards eventually get bulk-cleared by someone under pressure — which is strictly worse than not having a queue, because now the problem is invisible and someone signed off on it.

Measure the process, not just the day

Four metrics, tracked over time: match rate, exception count, mean exception age, and total write-offs. Their trends diagnose your upstream. A falling match rate means reference hygiene is degrading, usually because a new integration was added without the same discipline. Rising exception age means the queue is under-staffed. Rising write-offs mean something systemic is being absorbed rather than fixed.

This is the part teams skip and later wish they had not. A reconciliation system that only tells you about today cannot tell you that you are slowly getting worse.

The cultural half

Two rules that are not technical and matter as much as the pipeline.

Write-offs need maker-checker and a named reason from a fixed list. No free-text, and specifically no “adjustment” category. Adjustment is where drift goes to hide: a systemic problem gets recorded a thousand times as individually trivial decisions, each defensible, collectively material. The reconciliation engagement I ran was in large part the removal of exactly that category, and the write-offs that followed were far fewer because they suddenly required a sentence.

Daily, not monthly. Same argument as the drift article: one bad entry among three hundred same-day candidates is findable in an afternoon; among ten thousand month-end candidates, with context forgotten, it may not be findable at all. Daily reconciliation is not more rigorous — it is cheaper, because it is the only version where diagnosis is tractable.

Why this is the cheapest fintech discipline to retrofit

Because it requires nothing from the rest of your architecture. You do not need an append-only ledger, event sourcing or a rebuild. You need read access to your own records, the external files, and somewhere to put exceptions.

That is why it is always the first step I recommend to anyone already drifting. It does not fix the cause, but it stops the growth, and — more usefully — it measures the problem. Before reconciliation, “our numbers do not match” is an unbounded worry that cannot be funded. After a fortnight of daily matching, it is a categorised list with rupee values against each cause, which is a fundable work plan.

The sequence that follows — idempotency, then forensics, then the ledger core — is in the drift retrofit section, and none of it makes sense to start before reconciliation is running.

At national scale

The pattern holds at volumes most businesses will never see. India’s UPI runs past fifteen billion transactions a month (NPCI publishes the figures at npci.org.in), and the systems handling that share this architecture — normalised feeds, cascaded matching, reason-coded exceptions, aged queues. What changes at that scale is partitioning strategy, parallelism and the tolerance for manual review, not the shape.

That is genuinely encouraging if you are building at a smaller scale: the discipline you adopt now is the one that carries. The teardown series documents this across fourteen companies, including several operating at exactly that volume.

Building the reason taxonomy

This is the part that decides whether the system gets used, and it is a conversation rather than a coding task.

Sit with whoever currently resolves differences by hand and ask them, for the last twenty exceptions, why each one happened. You will get six to ten categories, expressed in their language, and every one of them will be a real pattern rather than a theoretical one. Encode those. Add an “other” bucket, monitor it, and when “other” exceeds a small share of exceptions, interview again — because a growing other means a new pattern has appeared that nobody has named yet.

What you must not do is invent the taxonomy from first principles at a whiteboard. The categories that matter are specific to your gateways, your bank, your settlement cycles and your product’s edge cases, and the people doing the work already hold that knowledge. Encoding it is the actual deliverable; the matching engine is the easy half. And treat the taxonomy as living: categories that stop appearing should be retired, because a stale list trains reviewers to pick the closest wrong answer rather than flagging something genuinely new.

Related reading

/blog/why-your-ledger-drifts — why late reconciliation causes drift · /blog/idempotency-done-right — preventing the duplicates you would otherwise reconcile · /work/reconciliation — the engagement, documented · /services/fintech — having it built