Platform Migrations
Data migrations are often treated as temporary work. Write a script. Run it. Fix the things that break. Run it again. Delete it when the project is over.
That approach works until the migration becomes large, the source and target systems are fundamentally different, and failure is no longer something you can afford.
For one major platform migration, I took a different approach.
Instead of building a one-off migration script, I built a source- and target-agnostic migration engine around an ETL pipeline.
The result was a reusable migration system capable of processing hundreds of thousands of records, validating data at multiple stages, recovering from failures and performing incremental migrations.
And it finished ahead of schedule.
The project involved moving data from a legacy Drupal-based platform into a legacy Symfony 1 platform.
The two systems had evolved independently, so this wasn’t a database copy.
The data models were different. Taxonomies were structured differently. Some source entities had to be transformed into completely different target structures. And some of the data — particularly complex CVs — required considerably more than straightforward field mapping.
The migration covered approximately:
- 50,000 users
- 10,000 jobs
- 500,000 CVs
- 300,000 job alerts
- Associated assets
- Supporting data
That scale changes the nature of the problem.
A failure at record 300,000 shouldn’t mean starting again from zero.
The first architectural decision was to separate the migration engine from the systems being migrated.
The engine itself knew nothing about Drupal or Symfony. Instead, the migration was described through migration profiles defining how a particular dataset should be processed.
The general flow was:
Extract → Validate → Transform → Validate → Persist
Each stage had a clear responsibility:
- Extract — read data from the source system.
- Validate — check that the extracted data met the expected assumptions.
- Transform — map the source model into the target model.
- Validate again — make sure the transformed representation was valid before touching the target.
- Persist — write the transformed data into the destination system.
This separation made the system much easier to reason about and, more importantly, made it possible to rerun individual stages without throwing away everything that had already worked.
One of the design principles was simple:
A migration will fail somewhere.
Not because the system is badly written, but because real data is messy.
There are malformed records. Unexpected relationships. Missing references. Files that don’t exist anymore. Data that doesn’t match the assumptions somebody made ten years ago.
So instead of treating failure as an exceptional event, I designed the migration to tolerate it. The engine kept track of its progress so that a failed run could be resumed rather than restarted from the beginning.
That made a huge difference at this scale. If something failed after hundreds of thousands of records had already been processed, we didn’t throw away the successful work.
We fixed the problem and continued.
Another important requirement was the ability to run delta migrations.
The initial migration was not necessarily the final migration. The source system could continue changing while the migration was being prepared.
Where the source data provided a suitable timestamp, the migration engine could identify records that had changed since the previous run and process only those changes.
This made it possible to move from:
Initial migration → delta → delta → final synchronization → cutover
rather than requiring the entire dataset to be migrated again every time.
Not every entity supported this approach — some datasets, such as job alerts and certain statistics, didn’t have the right characteristics — but where it made sense, it significantly reduced the amount of work required for subsequent runs.
The hardest part wasn’t moving the data. It was understanding what the data actually meant.
Taxonomy mapping was one example. The source and target systems had different representations, so categories and relationships couldn’t simply be copied across.
CVs were another. A CV isn’t just a row in a database. It can contain structured information, relationships, uploaded assets and nested data that needs to be reconstructed correctly in the target system.
The migration therefore became a transformation problem rather than a transport problem.
Extracting data is easy. Preserving its meaning is the hard part.
I didn’t want validation to happen only after something had already been written into the target system. That’s why validation gates existed at multiple points in the pipeline.
Data could be rejected after extraction if it violated source assumptions. It could also be rejected after transformation if the resulting target representation wasn’t valid.
This made errors much easier to identify and isolate. Instead of:
Something is wrong in the new system.
we could reason about:
The source data was valid, but this transformation produced an invalid target representation.
That distinction matters enormously when debugging a large migration.
The complete initial migration took roughly five hours. That’s not an impressive benchmark by itself.
But raw migration speed wasn’t the primary design goal. Reliability and recoverability mattered more.
The expensive work could be repeated safely. Subsequent delta migrations were considerably faster because they only processed changes. Asset processing could also be parallelized, which helped with the parts of the migration that benefited most from concurrency.
For a migration system, I’d rather have a five-hour process that I can confidently rerun and recover from than a two-hour process that requires everything to be perfect.
The migration wasn’t just a data exercise. There was a real platform transition at the end.
The engine allowed us to perform the initial migration, keep the target synchronized through deltas, and eventually perform the final synchronization before switching over.
That significantly reduced the amount of data that needed to be moved during the final cutover.
The result was a controlled transition rather than a single high-risk migration event.
The migration covered hundreds of thousands of records across multiple business domains and complex data structures.
The project was expected to be completed by the end of the year. We finished ahead of that expectation.
More importantly, the migration was completed successfully without having to build a collection of fragile, one-off scripts for every entity. The same underlying engine provided a consistent way to define, validate, execute and monitor the migration.
Once a migration becomes large enough, it deserves the same engineering discipline as any other production system.
“I didn’t build a script to move data once. I built a system that could safely move data as many times as we needed.”
This was one of those projects where the architecture was largely invisible to the people using the final platform. And that’s usually a good sign.
There was no flashy new frontend. No new framework for customers to see. Just a complicated migration that had to work.
The best part was that it did.
And it did so early enough that the migration engine itself became part of the solution rather than becoming another piece of technical debt left behind after the project.
