Data migration in the public sector has always had a certain theatrical quality. In housing especially, it feels less like a technical task and more like a high-wire act performed without a safety net, while an anxious audience of residents, regulators and colleagues watch from below.
For anyone involved in housing, the reasons are obvious. Housing data is sprawling, lived-in and full of history. It bears the marks of every policy change, system upgrade, manual workaround and well-intentioned spreadsheet intervention from the past twenty-odd years.
It’s rich, essential and occasionally mystifying, which makes lifting it from one system to another a job requiring patience, rigour and a staunch refusal to look down.
To those on the outside, the inherent complexity might be assumed to be due to the sheer volume, but in reality, the difficulty lies not in how much data there is, but the condition in which you find it.
Fields evolve, definitions drift and work-arounds proliferate in the wild. It’s all perfectly understandable in a busy housing provider, but it’s all of these small adaptations, done with the best of intentions at that point in time, which can derail a project’s ‘go live’.
Believe it or not, it’s entirely possible (and even likely) to come across two instances of the same system that are so different in their implementations that they are completely incompatible. This is a challenge faced by many after a merger or acquisition.
Migrating systems is arguably the most high-risk activity any large organisation can undertake.
There’s regulatory scrutiny, which doesn’t allow for excuses; there’s tenant impact, where a misplaced rent balance or a lost flag isn’t just an inconvenience, it can be a life-changing issue.
Then there’s operational reality – the actual moment you switch systems and the new one becomes the single source of truth. If that truth turns out to only be ‘mostly accurate’, you will definitely hear about it.
Method in the madness
While the risks are considerable, the real story is how predictable and manageable the whole process can become when it’s approached with a robust, repeatable method. Not something improvised on the day and not an heroic late-night dash. A calm, structured and trusted sequence of steps that can be repeated until the data behaves itself.
A reliable methodology brings method to the madness. It turns the initial phase, where everyone is politely pretending the data is probably fine, into an honest assessment of what’s actually lurking beneath the surface.
Through a detailed data discovery phase, patterns reveal themselves, oddities emerge and, for the first time, the full scale of the task ahead becomes visible. It’s the awkward but essential conversation before the point of no return.
Then comes the craft: cleansing, shaping, mapping, transforming and checking, over multiple cycles and iterations. Using each phase and each iteration to nail down what’s mandatory, what’s important and what’s no longer needed.
Each cycle tightens the rules, exposes lingering anomalies and brings the organisation a little closer to the point where it can say, with confidence, that the data is ready.
The complexity here is aligning the old with the new. Catering in the new world for the inconsistencies and indulgences of the old world, while trying to remain focused on the project’s original goal and aligned to the agreed timescales.
Headwinds are inevitable
Of course, even the best method faces a few familiar gusts of wind. Transformation programmes have a habit of suffering from moving goalposts as deadlines loom and the impending cut-over focuses minds on ‘what if…’ scenarios.
Data is intrinsically linked to every moving part of an implementation and fundamental to its success or failure, so even minor changes (and there are always, at least, some minor changes) can have significant knock-on impacts, which the data workstream is then expected to accommodate.
Business teams, whose knowledge is indispensable, are often brought to the table too late, after the rules have already been drafted and the data mapped. What was once set in stone quickly requires changing as the ‘reality of reality’ starts to bite.
Cleansing always takes longer than anyone wants to admit, mainly because it’s the point where long-standing operational habits meet the cold calculation of validation rules. Yet without it, any benefits the new system might bring are lost in the scramble to make good after the fact, when things have started to go wrong.
Keeping your balance
With a tried and tested approach, housing providers can weather these crosswinds more easily. Seasoned migration experts can see problems earlier, develop mapping and load routines capable of accepting short-notice changes, help plan for cleansing realistically and recognise the difference between a ‘mere’ nuisance defect and one that could cause trouble on day one.
Cut-over then becomes what it should be – a controlled and almost ceremonial moment where everything lands exactly where it should.
The real reward for all this care isn’t just a successful system implementation. It’s the calm that settles across the organisation afterwards.
Staff open the new platform and see data that makes sense. Tenants experience better, more consistent services rather than disruption. The senior leadership team avoids the dreaded ‘post-go-live’ crisis meeting and, in some cases, the need to rapidly update their CVs.
And the migration and implementation teams, usually exhausted by this point and looking for the nearest pub, can finally focus on the satisfaction of delivering the benefits the transformation programme was designed to unlock.
For those of us who work with this every day, the message is simple – migration isn’t glamorous yet nor does it need to be. It’s an essential component of any transformation programme, and a craft that rewards the methodical over the cavalier. Experience over improvisation, iteration over haste.
The tightrope will always be there, stretching from legacy to the future, but with the right preparation, the right partners and a method proven to keep organisations balanced and upright, the walk becomes less of a terrifying, tentative tiptoe and more a considered, confident stride.
David Bamford is the commercial director of Migra Data.
