Senior Software Engineer at Quandri
Data migrations don't have a compiler, so nothing tells you when a mapping is silently wrong. We built a score that does, and let an LLM write code against it until the score was good enough to keep.
Every engineer trusts the compiler. You rename a field, forget a call site, and the build goes red before you've tabbed back to the terminal. It's a tight feedback loop around your intent, and it tells you in seconds whether what you wrote matches what you meant.
Data migrations don't get that. When you map one data model onto another, nothing turns red when you silently drop a field, coerce a date into the wrong timezone, or land a value in a plausible-but-wrong place. The types check, the JSON is valid, and the tests you remembered to write pass. Months later, someone notices that an entire category of records has been wrong.
This is how we built something compiler-like for one of those migrations: a fast, automatic score that an LLM wrote code against until the score was good. We didn't train anything, and we never put a model in the request path. The model acted as a developer, and the score was its feedback. We call it a score rather than a loss because nothing here is trained, and the mental model we want is the compiler rather than machine learning.
A side effect we didn't expect was that the same loop became the toughest review our shared model had ever had.
Our automation bots run on an internal model we'll call IA. IA was built for the bots' convenience and tuned for iteration speed. It was never meant to be the contract between services, and for years, it didn't need to be.
That changed when we moved toward a service-oriented architecture, where services speak a deliberately designed, versioned, shared model we'll call UDM. To get our existing product onto it, we needed adapters in both directions: IA→UDM so the bot can speak the shared language, and UDM→IA so it can keep consuming data that now originates elsewhere.
We estimated it the usual way: about five weeks for someone who knew both models cold, closer to eight for someone who didn't. Both models are large, both are messy at the edges, and "correct" is spread across hundreds of fields and dozens of carriers in two countries. So we asked whether an LLM could do the grind.
The naive version is one prompt: here's a record, here's the target model, produce the output. It works the way these attempts always do. It handles the easy cases and invents answers for the hard ones, and you can't tell which is which, because the output is well-formed and looks right.
One-shot prompting fails here for the same reason a new hire fails on day one with an unfamiliar schema. They're both perfectly capable. Neither has any feedback telling them when they got something wrong. So we stopped trying to get the answer in one shot and built a loop that produces good ones instead.
This is iterative generate-test-refine, in the spirit of Self-Refine, with one change. The feedback doesn't come from the model judging its own work. It comes from an external score, computed by running the code over real data.
The loop has four steps:
Everything rides on step 3.
Scoring a mapping seems to need a correct answer for every record, a golden output hand-built by an expert that you diff against. Building thousands of those is its own multi-week project, so we sidestepped it. A faithful transformation shouldn't lose information, so we measured survival instead. Take every value in the input, take every value in the output, and count how many input values made it through.
@property
def value_preservation_rate(self) -> float:
if not self.input_values:
return 1.0
return len(self.preserved_values) / len(self.input_values)
Most of the work went into deciding what "the same value" means, so the score wasn't drowning in false alarms. "$1,234.00" and 1234.0 are the same number, "2023-09-13T00:00:00Z" and "2023-09-13" the same date, "ClaimStatus.OPEN" and "open" the same enum. Our normalizer is a stack of regexes, and it's blunt: strip the % and both $5 and 5% collapse to 5, so the number is an optimistic estimate rather than a proof.
The score has two limits by construction, and they decide how you're allowed to use it:
So we never treated the score as proof of correctness. We treated it as a fast signal with known holes, and we put a person on each hole. Every iteration the model kept a running PROJECT_CONTEXT.md: what's implemented, the current score, the assumptions it made, what's failing, and every field it had moved onto the exclusion list, with a reason. Every couple of iterations, a developer who knew the domain read it for about 10 minutes and did the thing the metric can't. That's not an acceptable exclusion, map it. That assumption is wrong, because a financing company is an additional interest, not a named insured. You're missing this field. Then they decided whether to continue or change direction. The developer never wrote mapping code. They guarded the denominator and steered.
We also ran the output back through the same model with a second, critique-and-fix prompt. It caught a few mistakes the score is blind to, but since it's the same model, it shares the same blind spots. It's a cheap self-check, useful but well short of an independent guard.
The score climbed from about 78.5% on the first working pass to 99.76% of in-scope values, meaning the values that remain after the exclusion list, over roughly 40 passes through the loop. That was two focused 10-hour days, an iteration about every 30 minutes, against a five-to-eight-week hand estimate.
The honest reading of "near-100%" is narrow. We preserve essentially everything we chose to carry; the exclusions are written down, and placement was checked by people rather than by a number. The adapter is in production today. Like any adapter, it meets new carriers and grows new edge cases, and we extend it as they appear. It is deterministic, though: same input, same output, no model in the request path, nothing to pay per record, and nothing to hallucinate at runtime. Determinism isn't correctness, and we don't pretend it is. A value can be deterministically placed in the wrong field, which is exactly why the score was never our only check.
We designed UDM months earlier and thought it was finished. Pointing a reality-checked loop at it with real data from two countries said otherwise, and said where. The score couldn't climb without changes to UDM itself, and each stuck point was a latent design issue surfacing: a reference field that needed to become an embedded record, a party slot that had to become polymorphic to hold both a person and an organization, and scheduled specialty items such as inland marine, which had no home in the model at all. None were obvious at design time. They became obvious the moment an automated check across thousands of records refused to let them slide, and we fixed them before they spread. We also came out with thousands of input and output pairs, a regression corpus that we didn't have before.
Most of this is old. Strip the LLM out and you're left with ETL source-to-target reconciliation, which compares values before and after a transform and which data teams have done for decades, and schema matching is a research field that predates LLMs by 20 years.
What's new for us is specific and small. That reconciliation score became the live signal an LLM author worked against, and what shipped was deterministic code. It's a variation on LLM code-migration loops. Airbnb's test migration is the well-known one, but they had a compiler and a test suite as the oracle. Data semantics have none, so we manufactured a reconciliation oracle instead. It also differs from self-improving schema-matching work such as Matchmaker, presented at a NeurIPS 2024 workshop, which improves an inference-time matcher from synthetic demonstrations and is measured on labeled benchmarks. We improve a deterministic artifact at authoring time against an unlabeled score.
The engineer still owns the judgment and the compiler still owns the code. What the loop compressed was the slow middle of the migration, the cycle of trying a mapping, finding it subtly wrong, and trying again, which now runs in minutes instead of weeks.
The current setup has an obvious next step. The score can't see placement and can't police its own exclusions, which is why a person guards both today. Moving that guard onto a second, independent model that audits the output and the exclusion list is where we're taking it next.
Mostafa Ahmadi is a Senior Software Engineer at Quandri, where he builds backend systems and automation solutions. He began his career as a data and backend engineer and holds an MSc in computer vision and deep learning, with around five years of experience across data and backend engineering. He cares deeply about good design, and will happily argue for a better one. He enjoys writing about on engineering craft, data and observability, and putting AI to work in production.