It’s often said that UK social housing providers are ‘data rich but insight poor’. The Social Housing Ombudsman stressed in last year’s Spotlight Report that there is a desperate need for the improved management of data, stating that it was “the closest thing the social housing sector has to a silver bullet”. Data-related risk is also now recognised as one of the primary threats to the sector. And yet, improved knowledge and information management also presents us with a considerable opportunity, the benefits of which will make the lives of tenants and those working in our sector measurably better.
A hidden opportunity
Within the vast expanse of data held in the sector’s housing and data management systems lurks a silent but significant component of this problem and opportunity – dark data. This term refers to the colossal amount of unstructured, untagged and untapped information that’s generated and stored by housing providers but not used to its full potential. To provide an idea of the scale, research by Gartner and IDC indicated that up to 80 per cent of an organisation’s data is dark, unstructured and not analysed and that almost 40 per cent of that data will have value once it has been analysed. Research also indicates that better management of dark data can reduce data storage costs by almost 60 per cent.
As our sector grapples with increasing demands for improved governance, efficiency and transparency, understanding and using dark data provides the key to unlocking a wealth of opportunities for improvement and innovation.
The accumulation of dark data is not without its challenges. Housing providers and local authorities often struggle with disparate data stores, legacy systems and complex data ecosystems. These lead to difficulties in maintaining data quality, governance standards and, ultimately, in making informed decisions that affect tenants’ lives.
Shining a light on dark data
By shedding light on dark data, organisations can gain insights into tenant demographics, housing conditions and improved financial management. This informs strategic thinking and is critical to addressing governance, risk and compliance challenges, thereby improving the overall quality of life for residents.
The new regulatory regime, introduced in April 2024, reinforces the importance of good data governance, leading many housing providers to focus their attentions on initiatives to tackle the issue of dark data. Indeed, 3C Consultants has seen a significant increase in requests for the development of information governance strategies and frameworks aligned with strategic objectives.
Focusing on TSMs
We’re seeing a particular focus on such areas as tenant satisfaction measures (TSM), health and safety, property disrepair and rent setting assurance, where enhancing data accuracy, completeness and consistency across different systems and data stores is critical. Critical components of the required data prove to be ‘dark’ and unstructured (i.e. data that’s not held in housing management software but in places such as spreadsheets, emails and shared drives).
If we take TSMs as an example, there is the need to determine the accuracy of data, often held in multiple data sources, and check the corresponding data flows in order to provide assurance around TSM calculations. Gas appliance management provides an example of health and safety assurance. In our experience, it’s not uncommon for the analysis of dark data to highlight properties where there are gas components but no record of servicing; where there are gas appliances but no carbon monoxide alarm; or where there are incorrect servicing cycles. Such analysis reveals inconsistencies between data stores, far too often due to spreadsheets that aren’t properly controlled.
The road ahead
As UK social housing continues to evolve, the role of dark data will become increasingly significant. Successful digital transformation strategies are reliant on the adoption of robust data management practices which, if implemented correctly, will not only streamline operations but also enable housing providers to deliver better services. Indeed, in our experience, transformation initiatives will fail to fulfil their potential unless data and information management transformation is a key component.
The journey towards illuminating dark data is challenging but the rewards of improved governance, enhanced tenant satisfaction and operational excellence make it well worth the effort, particularly when you also consider that the Regulator of Social housing has said that good data governance is critical to achieving the highest levels of regulatory grading.
Colin Sales is the CEO of 3C Consultants.
