The biggest danger in healthcare AI isn't that a model fails. It's that nobody notices when it starts performing differently.
Most healthcare AI systems don't fail dramatically. They drift slowly. And hospitals today are not operationally set up to detect that.
In our latest video, Ellie Asgari, kidney doctor at the UK's National Health Service and AI Scientist at Newton's Tree, explains how AI can drift without you even noticing - and what it takes to fix it.
Approval is not supervision
The uncomfortable reality is that most hospitals today do not have continuous operational surveillance for AI systems. They have approval processes. They often don't have supervision infrastructure. That's a very different thing.
In traditional healthcare governance, we're used to evaluating things at fixed points in time. But AI systems interact with live operational environments that continuously evolve. Which means the question is not: "Was this model safe at deployment?"
The real question is: "How do we know the system is still behaving as expected six months later?"
In many organisations, there isn't a good answer to that. Monitoring is still treated as optional. Or theoretical. Or something vendors vaguely claim exists somewhere in a dashboard. But operational monitoring is not a feature. It's an entire discipline.
What drift looks like in practice
Imagine a chest X-ray AI model deployed across a hospital network. Initially, performance looks strong. The validation was good. Clinicians trust it. Governance signs off. The deployment is considered successful.
Then over time, small operational changes begin happening:
- A scanner software upgrade gets rolled out.
- Different acquisition protocols start being used in certain sites.
- Patient demographics shift seasonally.
- A new workflow changes how studies are prioritised and reviewed.
None of these changes individually seem significant. And importantly - the AI still works. It still produces outputs. Nothing "breaks."
But six months later, the input data distribution no longer fully resembles the environment the model was originally evaluated in. Sensitivity may have dropped slightly. False positives may be increasing in certain cohorts. Maybe the model is now systematically underperforming on portable studies done overnight.
Nobody notices immediately because the degradation is statistical, not catastrophic. And that's the dangerous part. The most serious AI failures in healthcare are often not visible operationally. They emerge gradually in the background while trust in the system remains high.
How to fix it: visibility, thresholds and ownership
In practice, health systems need three things.
1. Visibility
You need to continuously observe how AI systems behave over time. That might include input data drift, changes in output distributions, subgroup performance, clinician-AI disagreement, operational usage patterns, and downstream outcomes.
2. Thresholds
Not every variation matters. The challenge is understanding what constitutes meaningful behavioural change versus normal operational noise.
3. Ownership
This is the part people underestimate most. Who is actually responsible for ongoing AI supervision once systems are live? Not procurement. Not just the vendor. Not just the data science team.
Operational AI governance requires defined accountability structures inside the health system itself. Because AI is no longer just software being installed. It's becoming part of clinical operations.
Deployment is not the finish line
The future winners in healthcare AI probably won't just be the organisations building the best models. They'll be the organisations that become best at supervising AI safely in real-world environments over time.
Because deployment is not the finish line. It's the start of continuous operational responsibility. And the most dangerous AI failure is often the one nobody notices happening.
Watch the full video here
This video addresses the "set it and forget it" approach to AI. Many teams treat AI like standard software, unaware that its performance can, and often does, shift and change as patient demographics and scanner hardware evolve. Ellie demonstrates what performance drift actually looks like in a clinical setting, how quickly it can happen, and the specific monitoring protocols required to catch it before it impacts patient care.
