Like many industries with roots in the public sector, who have historical data going back to the previous century, housing providers often find themselves struggling to maintain consistency across the range of data required for their day-to-day operations.
In some instances, housing management systems have grown organically over time and will be surrounded by ancillary systems, off-system databases and spreadsheets, meaning that operational processes have invariably become intertwined, causing complexity across many operational areas.
Data discovery
This presents a significant challenge, thankfully with a clear solution – data discovery. Data discovery has become a core part of any organisational transformation initiative, with the goal of structuring and implementing a single source of truth for all entities, with clearly-defined processes and models, structured to make the inherent complexities of the industry as simple as possible. Detailed data discovery should also uncover the location of documents, such as PDFs and Word files, that will be required to support business processes in the new world, an area that is often overlooked.
Data quality is also core to this process. The implementation of ongoing cleansing procedures and robust data-quality rules throughout the lifecycle of the ‘as is’ and ‘to be’ solutions has become crucial, in order to both measure the data shortfall and maintain the progress made throughout the process, holding those with the responsibility for data quality to account.
Data-quality issues introduced into the new solution as part of a migration often become more problematic to cleanse and continue to cause issues years after the migration has been finished, with the processes that introduced those errors lost to the previous system.
A robust data-quality process and strategy is critical to the success of the migration and, ultimately, to the transformation initiative as a whole.
Under the bonnet of the legacy landscape
Disparate legacy estates, often including the aforementioned off-system and ‘grey IT’ data sources, living side-by-side with housing management, customer relationship management (CRM) and asset management solutions, are likely to need to be supported by a complex web of automated and manual interfaces. Data is often maintained in multiple locations, frequently leading to cross-database integrity issues and duplication of data. Even within the same system and database table, there is often no clear answer to where the master source of a record lies. In certain cases, the master data source for an entity will differ attribute by attribute.
Such challenges lead to considerations for rationalisation and merging activities, commonly referred to as ‘golden record’ creation, whereby rules will be established to build an entity record from different locations based on a set of clearly-defined criteria and rules.
The complexity of such activities should not be underestimated. For complex data areas such as components, it’s not uncommon to need to start the golden record creation activities up to a year before the planned go-live date to ensure that the dataset to be migrated is accurate.
Tooling & migration architecture
It’s critical that robust technical infrastructure is established to support the migration process through the extract, transform and load (ETL) process, as well as the surrounding data quality and technical reconciliation activities.
Extraction routines will need to be implemented with careful consideration to BAU system processes and batches, and also to ensure that the security of data is maintained throughout the migration journey.
A robust migration environment will also ensure that accurate data quality reporting can be maintained even in the midst of a data-load cycle, which will require a static, point-in-time source dataset and code base.
It’s advisable to procure specialist data migration software, which should offer the ability to formally document and manage the data-mapping rules and logic, define data-quality rules and report on their output, and also support automated data profiling.
Migration preparation
As well as understanding the scope of the data migration in terms of source systems and data entities, the volume of data to be migrated also needs to be understood and defined in detail.
It is rarely advisable to migrate all records from source to the new target system(s). As the volume of data increases, so do the cleansing efforts and reconciliation activities, as well as the time required to process new records into the target environment in each load cycle (not to mention the impact on the new target system’s performance, invariably affecting the user and customer experience after the go-live).
Careful consideration should be given to data retention rules and policies, as well as regulatory reporting requirements. Future-state enterprise reporting capabilities will play a key role in determining the approach for the historical data that should not be migrated, and the obfuscation of personally identifiable information (PII) and sensitive personally identifiable information (SPII) to allow data science teams to perform trend analysis should also be considered.
In general, migrating the bare minimum of data required to support the operational business processes is a sensible approach.
Prior to each data-load cycle, there will be a key dependency on the preparation of not only the master data and transactional data entities, but also the reference data and configuration values that will be used by the new target system. The time and effort required to define these is often overlooked, and the mapping and development of master data and transactional entities is reliant on their delivery in plenty of time for the first load cycle.
For the new-age of highly-configurable software, it’s not uncommon for the number of target reference data and configuration entities to number into the hundreds, the definition for each requiring a rigorous approach to ensure that the target ways of working can be achieved. This requires input and ownership from the subject-matter experts (SMEs) with in-depth knowledge of their respective business areas, as well as support from the solution experts to advise on functional usage and impacts.
Conclusion
Unfortunately, due to the above complexities, migrations can often be as harmful as they are helpful. Few would dispute the requirement for the implementation of more modern and flexible systems within the housing sector, the requirements for which often arising from constraints within legacy systems that have required them to be altered over time to suit the changing industry. While good solutions in the short-term, these alterations often add complexity and convolution to operational processes and system maintenance activities, as well as leading to data duplication.
The implementation of new systems to help rationalise the legacy system landscape offers a way to remove these complications, but only if this ultimately allows the business to cease operations within the previous system completely. Sadly, this isn’t always the case and the organisation can find multiple systems working in parallel for a period of time.
Instead of the new ‘single source of truth’ solution they set out to attain, housing providers often then find themselves living with a more complicated, transitory architecture, further fragmenting their operational processes and introducing technical debt for tactical integrations.
By establishing a clear strategy and adopting tried-and-tested methodologies, technical facilitators and expert consultants, the risk of a failed migration can be eliminated. The decision, however, is up to you.
Being aware of these common problems in advance will really help the data migration delivery be a success. Unfortunately, with migration there are no short-cuts; they just lead to greater risk, unexpected delays and higher costs.
Ben Charlton is head of migration at IntoZetta, and Daniel Thompson is a senior data consultant at IntoZetta.
