Over recent years, local government has increasingly begun to adopt a more ‘data-led’ approach to tackling some of the most pressing challenges. But how do you go about effectively delivering a data product?

In this blog, Anna Humpleby, LOTI’s Data Projects Manager, and James MacTavish, Engagement Manager at Faculty AI, share insights into a tried and tested process for developing a data product, based on the development of the Pan-London Strategic Insights Tool as part of LOTI’s Rough Sleeping Insights Project, and some key guiding principles you can apply to your own data product development.

How did we conduct an effective discovery?

James: For the Rough Sleeping Insights project, we conducted discovery over a 7-week period consisting of 3 key steps:

• Understanding the problem space and ecosystem:

We established a clear understanding of the rough sleeping ecosystem and the problem space by building robust relationships with prospective users and key stakeholders, conducted 1:1 user research sessions and workshops, and defined user needs and journeys.

• Understanding the data:

We requested samples and schemas of the data sources available for the project to conduct analysis that enabled us to build an understanding of the data, including its quality and characteristics.

• Developing initial designs and non-functional system requirements:

Through iterative user research and close collaboration with technical teams, we developed the initial designs and defined non-functional system requirements to guide the SIT infrastructure development. Non-functional system requirements are the criteria that define how a system should work, rather than what it should do. For example, defining the security requirements for the SIT and building that into the system architecture designs.

What are the technical steps involved in building a data product?

Anna: In this project we conducted iterative development work over a 6-week period to produce a Minimum Viable Product (MVP). The MVP approach essentially involves creating a basic version of the product with a core set of features that meet pre-defined user requirements and deliver value. It allows you to quickly test viability with users and gather feedback for further development without investing too much time, money or resources into a product that ultimately may not meet users needs. 

James: The MVP build also involved 3 key steps:

• Data matching model development

We explored and tested various data matching approaches, ultimately selecting Splink (The Ministry of Justice (MOJ)’s open source data linking package) based on its performance and relative trade-offs.

• Infrastructure build

Concurrently with model development, we designed and built the necessary infrastructure. This included developing the necessary data pipelines with source systems, and a bespoke single data model to enable mapping (a key step in the data matching process).

• Minimum viable product launch

We successfully deployed the initial minimum viable product to the pilot group of four boroughs and four service providers.

How can you successfully test an MVP with users?

Anna: It’s really important to test the product with users and they should really try to use it more than normal during the testing period. For example, if they plan to use it once a week normally, ask them to use it 3-4 times a week during the testing period. A user testing phase should be more intensive as users try to both test and ‘break’ the system or process. It’s also really helpful to provide a set of guiding questions to prompt users as they experiment with the product. System testing is also necessary to allow you to meet your obligations under data protection law, if your system or process uses personal data (as the Strategic Insights Tool does). LOTI has developed guidance on information governance (IG) considerations for system testing.

James: As well as the user testing, we provided training. This included developing a user guide as well as delivering tutorials on how to use the tool via Microsoft Teams and a video demo for users to watch offline. The team also worked on investigating and debugging any issues reported by users during the testing period. Finally, we put together a roadmap for future system development based on the user feedback collated.

What comes after the MVP?

Anna: Wider rollout and further development! As we ran the pilot with just four boroughs and four service providers, ‘phase 2’ of the project meant rolling out the tool to the remaining 29 London boroughs and an additional selection of service providers. In practice this involved:

• Reviewing information governance arrangements 
• Engagement and briefings with relevant stakeholders across the boroughs and service providers (including rough sleeping practitioners, data leads and IG leads)
• Onboarding of data from those organisations
• Enabling access to the tool for new users

A crucial step of this part of the process was conducting an information governance (IG) review. The original IG documentation was developed before the product build, so had little detail on the system itself, and was designed only for the pilot phase, which included just 9 organisations as opposed to Phase 2 which would see this increase to 42.

Conducting an IG review is recommended practice for data sharing projects and should be a key part of your project/product management process. Consult LOTI’s guidance on how to conduct an information governance review for support on doing this yourself.

In your view, what has been the key success or enabler for this project?

Anna: Our process enabled us to establish strong data sharing relationships across local government and the voluntary sector, enabling join-up of previously siloed data sets for the first time, offering completely new insights to users. This has gained us international recognition and interest in this project. In January 2025 I presented this work at the International Journal on Homelessness Conference (IJOH), and in February 2025 James and I showcased the project at the AI Action Summit. We were 1 of just 50 projects selected from 770 applications across 111 countries, and 1 of 10 selected to pitch!

James: This project would not have been possible without LOTI’s information governance expertise and input. They worked incredibly hard before we kicked off in June 2023 to secure all of the necessary approvals to enable data sharing for the pilot phase. We worked very closely with the LOTI team to then secure those same approvals from each borough and service provider ahead of the full roll-out between September 2023 – February 2024. This dedicated workstream was, in my view, the single biggest enabler that was critical to the project’s success.

What was the biggest challenge to overcome?

James: Despite rough sleeping being a growing social issue, we were generally working with relatively low numbers. We had to find the right balance between providing meaningful insights to users and protecting the privacy of individuals. Once we had developed the user interface, we were able to identify the visualisations that put certain groups at risk of statistical disclosure (i.e. a scenario in which a number is so low that one can infer the identity of an individual through the characteristics that are represented through a particular visualisation / set of filters). We agreed that it was sensible to redact any data that described a group of five individuals or less, thereby enhancing the SIT’s privacy controls. We tested this solution with users to ensure that visualisations remained usable whilst simultaneously ensuring we protected people’s privacy.

What is one main takeaway from this project that you’d like to share with others?

Anna: We’ve gained insights far beyond just those presented to users in the visualisations available to them through the tool. Bringing together this data and the analysis around it has surfaced some really important challenges around processes and practices in the sector, and we’re seeing that individuals experiencing rough sleeping and homelessness may not be getting through the door as much as expected, or are not being recorded in the relevant systems.

What’s next for this project?

Anna: The project has really helped us begin to understand how people move between the streets, Housing Options and accommodation, but we know this is still only a small part of the picture. We want to expand the scope of this work away from looking at rough sleeping in isolation, but instead address homelessness in its widest sense.

We will continue to be outcomes-led in the next stages of development. Front and centre for us is looking to the sector to surface the key questions they’re asking: what is it that they need to know or be able to do that they can’t currently? And how might we develop the strategic and technical capability to answer them, leveraging the huge amount of data that already exists.