Figure 1. The governance blind spot.
Patient safety has understood for decades that error is not a character flaw. It is a system property. To Err Is Human built the modern patient safety movement on that insight: that human error is inevitable, and that clinical systems — from critical incident reporting (CIRS) to the governance frameworks of clinical risk management — must be designed to absorb, surface, and learn from it. The entire architecture of patient safety assumes one thing: that the human remains in the causal chain, visible, interruptible, and ultimately accountable.
Agentic AI inverts the problem. The question is no longer what happens when the human errs. It is what happens when the machine errs or when the interface is designed in a way that makes human error not just likely, but structurally guaranteed. The Swiss cheese model assumed fallible humans and defensive systems. Agentic AI introduces a third condition: a system that performs well enough, often enough, to erode the vigilance that safety depends on. That is not a technology problem layered onto an organisational one. It is a categorical mismatch between the architecture of the tool and the architecture of the institution deploying it. This holds across the spectrum, whether systems are fully autonomous, semi-autonomous, or bounded by explicit constraints. The governance gap is not a matter of degree. It is structural.
Procurement is where this mismatch becomes concrete. AI acquisition decisions are made through processes designed to evaluate equipment — assessed on cost, compliance, and vendor credibility — not on clinical workflow fit, accountability mapping, or governance architecture. Innovations are acquired, deployed into workflows that were never redesigned to receive them, and handed to clinicians whose job descriptions, scopes of accountability, and professional standards have not changed. The result is not transformation. It is complexity without coherence.
“When deployment outpaces integration, technology does not resolve fragmentation. It compounds it.”
The human-in-the-loop is not a safety net.
On a Monday post-take round, an agentic medicines-optimisation system presents the covering doctor with forty-one prescribing actions across the night’s admissions: reconciliations, renal dose adjustments, and several deprescribing proposals. Each carries a green confidence marker and a single Approve control, with the underlying reasoning two clicks away, behind data she has neither the time nor the full visibility to reconstruct. She clears the queue in under four minutes, because the round is moving and the system has been right almost every time for six months. One action continues an anticoagulant at a dose the agent inferred from an outdated weight; no one re-derives it, because the interface was built to be cleared, not interrogated. The approval is recorded against her registration number.
This is the design working as intended, not an aberration. We have built systems in which a human ratifies a decision they did not make, on information they cannot fully interrogate, in a timeframe that forecloses real evaluation. That passes for oversight without being it; more dangerous than full autonomy or full human control, because it carries accountability’s legal form without its substance. The fault is structural. These systems run human-on-the-loop: the clinician monitors but does not gate each action, while regulatory and professional frameworks still assume human-in-the-loop, placing responsibility on a named individual presumed to have authorised the act. The signature at the end of the queue closes that gap on paper and nowhere else. When the rare error surfaces, liability has already gone to the person least able to have caught it — the out-of-the-loop reviewer whom the low error rate has made least vigilant (automation bias).