LOTI is starting a new research project at the intersection of our AI (Artificial Intelligence) and Data Ethics work, critically examining ‘humans-in-the-loop’ (HIL), the humans in our councils who provide oversight of the AI and automated systems that are becoming part of local government. This is a research project that we hope will lead to practical advice and recommendations for councils and other public sector organisations who are designing public services that use AI, algorithms or automation, and want to understand how to govern them responsibly. 

Hanna Barakat & Cambridge Diversity Fund / Better Images of AI / Turning Threads of Cognition / CC-BY 4.0

As the use of AI increases in local government, the question of how humans are or should be involved in the decisions and services that we design become pertinent. In LOTI, we have seen an emerging trend in which councils designate an officer as a ‘human-in-the-loop’ for an AI or automated system. This role typically involves a service-level officer who evaluates and reviews the output of an AI system, subsequently correcting, approving, or using it to inform decisions. In theory, the presence of this role is both a bulwark against a system being considered fully automated, and this officer also acts as a safeguard against AI error. This would allow councils to access the benefits of technology, such as enhanced efficiency and accuracy, while mitigating potential risks and harms.

However, my readings over the past few months have raised some concerns with this logic. I was significantly influenced by Cory Doctorow’s 2024 blog, which referenced Ben Green’s 2022 paper, ‘The Flaws of Policies Requiring Human Oversight of Government Algorithms.’ Additionally, Sebastian Tschiatschek and Vienna University colleagues’ 2024 paper, ‘Challenging the Human-in-the-loop in Algorithmic Decision-making,’ provided valuable insights into the practical dimensions of ‘human-in-the-loop’.

Why is this research important?

This research is particularly timely in the context of the Data (Use and Access) Bill, which introduces changes impacting Automated Decision Making (ADM). The Local Government Association’s briefing highlights a crucial modification:

“Under new article 22A, a decision would qualify as being “based solely on automated processing” if there was “no meaningful human involvement in the taking of the decision”. A decision would qualify as being a “significant decision” if it produced “a legal effect for the data subject” or had a “similarly significant effect for the data subject”. New article 22D would give the secretary of state the power to make regulations to describe cases that are or are not to be taken to have meaningful human involvement, or what is or is not to be taken as a significant decision.”

Therefore, defining “meaningful human involvement” is a critical issue that this project will partially aim to address by providing guidance on what should constitute “meaningful human involvement”.

There is another issue, which I think is more fundamental from an ethical perspective. We simply don’t know how effective HILs and human oversight of AI or algorithms in general actually is in mitigating ethical risks. Should this be considered a necessary or sufficient component in the responsible governance of AI and automation? If so, what are the necessary conditions for this to be successful as a governance mechanism?

The following represent some of the possible critical issues that we also need to better explore and understand, as part of our mission to create practical guidance for public authorities on this:

• Lack of accountability for the human: If the HIL role lacks defined parameters, it cannot be considered meaningful governance but it cannot be held accountable to any expectations.
• Organisational factors make oversight impossible: Insufficient skills, time constraints on officers, organisational pressures (finances) or KPIs (key performance indicators) could lead to poor decisions.
• Technical factors make oversight impossible: Lack of AI transparency could hinder effective evaluation if the HIL lacks the required information about how an AI output was created came to a decision.
• Human oversight can still mean a system is considered automated: Real human oversight but which does not meet “meaningful” standards or conditions might still therefore constitute automated decision-making.
• Legal liabilities shouldn’t fall with frontline staff: It may be unfair to hold frontline staff liable for checking AI systems. But who is liable if the HIL fails?
• Human oversight is undermined by human flaws: Human biases or flaws could actually be introduced by having a HIL, making a technical system less fair or accurate.

Nonetheless, we may simply want a human involved in spite of some of these tensions within the concept. For one, humans at least can be held accountable for their mistakes in a way that a model like LLM (large language model) can’t. There may also be some services where we want HILs because we value the social relationship between an officer and resident, or because conversations with a resident allows councils to gather information about their local place and people might not. It may also be that residents simply prefer humans making decisions even if they are worse and more expensive than AI. Ultimately, there is a chance that by removing humans from public services that we are slightly eroding at an implicit foundation of the social contract between humans and councils. 

All of these ethical considerations should have implications for how we use AI in the future. The great risk, as I see it, is that the concept of HIL offers what Ben Green called a “a false sense of security”, under which organisations start using more risky AI products or automated systems believing that those risks have been mitigated by an HIL. So in practice we could be taking on more risk by avoiding actually addressing the underlying concerns.

Our research plans

So our inquiry is to fundamentally better understand the dimensions of how humans and ‘machines’ should co-exist in our organisations. I want to understand what those conditions are: what are the roles that humans should be playing or will naturally exist in, what are the skills that different people will need, how do we manage these people, who do we hold accountable in situations where different types of decisions are made with AI or automation or algorithms in conjunction with humans?

Our current research questions – which may evolve – are:

1. How effective can humans be at mitigating risks of AI, automation and algorithms?
2. What are the relevant critical dimensions to understand if a Human in the Loop does constitute ‘meaningful human involvement’?
3. How should local authorities design and account for ‘Humans in the Loop’?

Our research plan includes:

• Desk research of academic and legal literature;
• Workshops with local government and external experts;
• Interviews and case studies of HIL in local government;
• A final report with practical resources and recommendations targeted at local government;

If you are interested in this topic and want to contribute to our research, or have any particular examples of how you might have accounted for ‘Humans-in-the-loop’ then please don’t hesitate to get in touch.