Stop Measuring Migration Success on Day One.

There is a number you’ve likely run across if you’ve ever scoped an enterprise data migration: 83% of migration projects fail, overrun their budgets, or bust their schedules. It is almost always credited to Gartner, and it is a staple of standard vendor marketing.

But as engineering leaders, we need to be precise about our telemetry.

The 83% figure is old. It is traceable back to an early white paper published by Oracle and LumenData, and it gets repeated far more often than it gets audited. While it isn’t entirely wrong—migrations are notoriously high-risk—it tells you that projects fail without explaining the underlying failure mode.

The better-grounded, quieter numbers tell a much more useful story for modern infrastructure teams:

• The Cost of Mismanagement: McKinsey metrics show that the average migration runs roughly 14% over its planned cost, contributing to more than $100B in wasted global spend over a three-year period.
• The Operational Reality: The Bloor Group puts the outright project failure rate closer to 38%, with cost overruns averaging ~30% and schedule delays hitting ~41%.

Notice that even the grounded, audited figures are bleak. The core problem isn’t that moving data across a network pipe is difficult. The problem is what happens when the logic you used to move it turns out to be wrong.

The Failure Mode Hiding Behind a Green Dashboard

Most teams plan a migration as if it were a one-way trip. They spend months defining transformation rules, hand-code the pipeline, run the payload, and point to a high field-completion rate as proof of victory.

This creates a dangerous illusion: quality degrading quietly behind a green dashboard.

Consider a real-world example from a published CRM migration teardown. A company migrated its data into a brand new, $2.3 million enterprise platform. On paper, the migration was a success. In reality, a single “opportunity stage” field that should have contained exactly 6 standardized enterprise values was holding 89 distinct variations. A single typo—“Closeing Soon”—was faithfully carried over into more than 1,200 active records. The same system imported 247 different industry classifications instead of the mandated 15.

The platform worked perfectly. The pipeline executed cleanly. The payload was a complete mess.

When field-completion percentages look healthy simply because fields are full of inconsistent nonsense, the metric is theater. Completion measures whether data moved once under your first set of assumptions. It tells you nothing about what it will cost you when you discover those assumptions were wrong.

The Real Core Metric: Cost of Correction

After twenty years of watching these infrastructure projects play out, the pattern is entirely consistent: nobody gets the transformation rules right on the first attempt.

Requirements change mid-flight. A field gets re-interpreted three weeks after kickoff, a business merger drops an unplanned secondary source system into your scope, or a compliance officer asks for the target data to be shaped differently.

The moment that happens, your existing hand-written logic is wrong. In a conventional setup, fixing it requires a data engineer to manually unwind hundreds of fields of custom transformation code—often against a legacy source system that is already being retired. Because each redo is slow, expensive, and introduces regression risks, teams choose to patch and work around the problem instead of fixing it properly.

An agile migration model inverts this entire dynamic. It accepts that rules will change, and focuses on one architectural priority: making the cost of correction cheap.

To build a migration pipeline you can actually trust, your architecture must be able to answer three questions instantly:

1. What rules are we applying? Validation, transformation, and de-duplication rules should live as a readable specification, signed off before a single row moves. You cannot safely revise rules that are buried invisibly inside custom code.
2. What happened to every record? You need definitive telemetry on the rows that failed validation, the duplicates, and the broken fields. A system that silently drops inconvenient records to keep the completion percentage looking clean is an operational hazard.
3. Can we undo it fast? When requirements shift, can you roll back to the preserved original data, adjust the logic, and re-run the entire dataset in minutes, or are you stuck rebuilding pipeline logic by hand?

A Predictable Methodology

The technical cost of changing your mind should be near zero. At Consuly, we broke this down into a simple, reproducible methodology:

• Rules Before Rows: Stop discovering your data errors mid-flight under deadline pressure. We use modern AI for exactly one job: to analyze a representative data sample and draft the validation and transformation rules as a human-readable JSON specification. AI authors the spec fast; humans approve it.
• Review Before Run: Once the spec is approved, deterministic code executes the ruleset against the full dataset. Every manipulation, duplicate, and quarantined error is logged as a continuous, explainable chain. No black boxes, and no silent drops.
• Reverse on Demand: By preserving the original source dataset alongside an unalterable manipulation log, the entire mapping becomes fully reversible. If a rule turns out wrong, you don’t patch code—you roll back, adjust the spec, and re-run the pipeline deterministically in minutes.

The technical complexity of a data migration shouldn’t punish learning. When redoing a mapping becomes fast and safe, a wrong rule stops being a project crisis and simply becomes a normal, expected iteration.

Sources

• [1] Oracle & LumenData: Put Your Data First or Your Migration Will Come Last (Gartner 83% analysis origin). Source link
• [2] McKinsey & Company: Why Do Most Digital Transformations Fail (Data on migration cost overruns and global transformation waste). Source link
• [3] The Bloor Group: Running Out of Time (Data migration failure rates, schedule overruns, and budget overruns analysis). Source link
• [4] William Flaiz: Why Your Platform Migration Failed (CRM data-quality teardown and field-completion analysis). Source link