Zero-Downtime Migration

Migrating a live system without stopping it is a solved problem with an unforgiving recipe: expand-migrate-contract schema changes, dual-writes behind a flag, backfill in verified batches, parallel-run until the difference holds at zero, then a cutover that is a checklist rather than a heroic weekend.

Why the freeze is the real problem

Every failed migration plan I have read begins the same way: first we freeze changes. And every business that needs a migration cannot freeze — revenue flows through the old system hourly, and the product backlog will not wait for a rewrite.

So the plan is postponed, and postponed, while the legacy system compounds interest as both the largest risk on the books and the most reliable excuse for inaction. “We can’t touch that” becomes an accepted constraint rather than a problem with a method.

The method exists. It costs discipline instead of downtime, and every step of it is designed around one property: retreat must stay cheap.

The recipe

Expand, never mutate

New schema lives alongside old. Add columns and tables; never repurpose in place, never rename something the old path reads. The old code must keep working until the exact moment it no longer has to.

This produces temporary ugliness — two representations of the same concept coexisting — and that ugliness is the price of reversibility. Contract later, once the old path is genuinely dead and you have proof.

Dual-write behind a flag

Writes hit both worlds. The flag means retreat is seconds away rather than a deployment away, and that single property is what makes teams brave enough to proceed. Engineering the retreat is engineering the advance.

Dual-writing has one subtlety worth naming: the two writes cannot be atomic across systems, so you must decide what happens when the second fails. The usual correct answer is that the old system remains authoritative during the parallel phase — its write succeeds or the request fails — and the new system’s write failures are logged as discrepancies for the comparator to catch. That keeps the blast radius on the side you are not yet trusting.

Backfill in verified batches

Years of history move in chunks, each reconciled before the next departs. When the inevitable ancient data quirk surfaces — and it will, deep in the archive where old code wrote states the current schema considers impossible — it costs a batch’s investigation rather than a weekend’s rollback.

A thousand verified batches beat one optimistic bulk-insert every single time. The bulk insert is faster when it works and catastrophic when it does not, and it always does not, because 2019 contained decisions nobody documented.

Parallel-run to diff-zero

Both systems answer every query; an automated comparator diffs the answers daily and reports discrepancies. Each one gets chased to root cause, and each one is a production incident that never happened.

Two things matter here. Compare answers, not tables — the schemas differ by design, so you compare what the business asks: balances, positions, totals by dimension, individual lookups. And hold diff-zero long enough to cover every periodic process, which for a money system means at least one month-end, because month-end code paths are precisely the ones nobody remembers exist until they run.

The comparator becomes the project’s real manager: silent, unbribable, and the only stakeholder whose sign-off means the thing is actually done.

Cutover as anticlimax

Flip reads, watch, flip writes, watch, retire. A Sunday morning, a checklist, and no adrenaline. If cutover feels exciting, a step was skipped — the excitement is information.

Where it goes wrong

The undocumented scheduled job. There are always about three, each known to one person, each load-bearing. One of them is why the inventory step comes first, and why the inventory has to be built by reading the system rather than by asking people what it does.

Sequence and ID collisions. Decide ownership early: either the new system generates and the old consumes, or you use non-overlapping ranges. Discovering a collision during the parallel run is a Tuesday; discovering it after cutover, with references already given to customers, is a different kind of day.

The manual fix applied to one side. Someone corrects a record directly in the old system during the parallel phase — reasonably, under pressure — and the comparator lights up. Now you cannot distinguish real divergence from a known intervention. The fix is procedural: during a parallel run, manual interventions are applied to both sides or recorded as expected discrepancies, and everyone touching production knows this.

Trusting a clean comparator too early. A comparator that finds nothing in its first week may be measuring the wrong thing. Test it deliberately: introduce a known discrepancy in a staging copy and confirm it is caught. A silent comparator and a correct system look identical from the outside.

What this buys beyond the migration

The cultural change is larger than the technical one. Once a team has watched a live migration land safely, the next one gets planned rather than feared — and “we can’t touch that system” leaves the vocabulary. That vocabulary shift is worth more than any single migration, because it unblocks the decisions that were quietly routing around the legacy system.

I have run this on a financial ledger with years of history (the migration case study) and on a factory floor moving off a decade of registers (the ERP version). Different domains, same recipe, same anticlimactic Sunday. The method transfers because the constraints are structural rather than domain-specific: history matters, stopping is not an option, and confidence has to be earned by measurement rather than argument.

The honest cost

This is slower than a rewrite-and-switch, and it should be described that way rather than sold as free. You build a bridge you will later delete, you maintain two paths for weeks, and you spend real engineering time on a comparator that produces no features.

What you buy is the elimination of the failure mode that kills migrations: the one where you are half-migrated, out of executive patience, and running two systems permanently. That outcome is strictly worse than never starting, and it is the default outcome of the fast approach on any system with real history.

The inventory step, in detail

Everything above depends on knowing what the old system actually does, and that is harder than it sounds because the answer is not documented and the people who know it will describe intentions rather than behaviour.

Read the artefacts instead of interviewing. Every table, with row counts and last-write times — a table nobody has written to in two years is either dead or seasonal, and the difference matters. Every consumer, found by reading connection logs rather than by asking, because integrations exist that nobody remembers authorising. Every scheduled job, from cron, from the scheduler, and from the application’s own timers. Every trigger and stored procedure, which is where business logic hides in older systems and where a migration that only moves data will silently drop behaviour.

Then confirm the inventory by turning things off in staging and seeing what breaks. An inventory nobody has tested is a list of assumptions, and the assumptions are exactly what the migration will trip over — usually at the least convenient hour.

Related reading

/work/migration — the method on a live financial ledger · /work/erp-migration — the same discipline on a factory floor · /services/delivery — having it run for you · /blog/postgres-in-production — schema changes under load