Pilots conducting pre-flight procedures
getty
An ambient AI scribe, a program that listens to a patient-clinician conversation and generates a clinical note, documented that a medicine had been prescribed to treat post-traumatic stress disorder. However, neither the medicine nor the diagnosis were discussed – this was a complete hallucination.
The note never reached the patient because it was caught before it left the clinic.
Child and adolescent psychiatrist Dr. Jennifer Shannon, Co-Founder and Chief Medical Officer of Glacis, reviewed the transcript directly and concluded there was nothing in the visit that could reasonably have produced the statement.
“Imagine managing 20 patients a day, each with pages of generated notes,” she told me. “Defaulting to human-in-the-loop as your protection is not enough. Humans are fallible, busy, and often overtired.”
Documentation burden is real. Nobody seriously argues that clinicians should spend their evenings clicking boxes and typing notes. But implementation changes roles in ways that are easy to miss. A clinician who once distilled their understanding from notes taken during a visit is increasingly reviewing machine-generated summaries and structure derived from that same encounter. Responsibility hasn’t changed, but the nature of the work has.
I don’t picture attentive reviewers sitting quietly examining every sentence. I picture Tuesday afternoon. Somebody running behind. An inbox with thirty messages. A kid needs to get picked up. A patient waiting in room three.
Reality is messy.
Expertise Is Part Of The Safety System
Dr. Richard Rieck is both a practicing neuroradiologist and pilot; he is used to operating in multiple high-risk environments.
Pilots spend enormous amounts of time in simulators where multiple failures are intentionally layered on top of one another. Automated systems fail. Engines fail. Weather changes. The goal isn’t realism; the goal is to sharpen skills and build instincts before any of those situations occur in real life.
As Dr. Rieck put it, “After enough simulator training, flying the actual airplane almost feels like a relief because everything just works. You train the failures so that when something unexpected happens, you don’t have to invent a response.”
In aviation, all of these training and skill protocols exist because modern aircraft and their automated systems are extraordinarily reliable. Nobody boards an airplane expecting the pilot to disappear. Maintaining expertise isn’t evidence that the automation isn’t good; maintaining expertise is itself part of the safety architecture.
Automation Changes The Work More Than The Accountability
Dr. Richard Rieck thinks about AI in radiology in similar terms. Just as pilots understand both their own failure modes and the limitations of automation, radiologists increasingly need to understand where they are likely to make mistakes and where AI systems are likely to make mistakes.
Neither is perfect. The safety comes from understanding where each is most likely to fail.
I keep returning to this because “human-in-the-loop” has become one of AI’s favorite safety concepts. The idea is straightforward: before AI-generated work reaches a patient or customer, a person reviews it and catches anything that shouldn’t be there. Too often, though, we describe that person as little more than a backup. The machine does the work and the human corrects it when something breaks.
That doesn’t really describe what I see.
The people aren’t there simply to rescue the machine when something goes wrong. Their experience, instincts, and judgment are part of the system itself. As automation improves, those capabilities become more valuable, not less.
The Most Overlooked Safety Layer? Patients.
High-risk industries rarely rely on a single safety layer. Healthcare doesn’t either. Clinicians aren’t the only people whose expertise helps keep the system safe. Patients have quietly developed expertise about their own care for years.
Long before ambient documentation arrived, patients were already reconciling medication lists, catching referral failures, correcting demographic mistakes, and trying to make sense of conflicting recommendations from different specialists. They weren’t simply participating in care—they were identifying failures inside a system they could only partially see.
Patient advocate Hugo Campos, who alongside Liz Salmi has written about critical AI health literacy, frames this through a distinction between institutional AI and patient-directed AI.
“We’ve got to stop the system from being the gatekeeper, and we have to allow for people to help themselves,” he told me.
His point resonates because AI shouldn’t diminish patient expertise; it should strengthen it. As clinicians adapt to automation while managing overflowing inboxes, documentation burdens, and constant time pressure, patients become an increasingly important source of resilience within the system—not because responsibility has shifted to them, but because their perspective catches failures that no one else can fully see.
Professionals Deserve A Playbook
Most conversations around AI risk focus on whether the machines will fail. I increasingly wonder whether another question deserves more attention.
What happens to humans after years of successful automation?
Pilots train failures before they experience them. Sports teams rehearse situations they hope never happen. Trauma teams rehearse codes. The goal isn’t to eliminate surprises; it’s to avoid asking professionals to invent solutions under pressure.
Honestly, some of these systems still leave me shaking my head. They’re astonishing. Which is exactly why I worry about what happens to human expertise when they work correctly most of the time.
AI systems are getting very good. That’s exactly why I find myself thinking about skill decay more than catastrophic failure.
Nobody wants to discover skill decay in the middle of an emergency.
And no professional should have to invent a response when the stakes are highest.
Now is the moment to be intentional about the skills we continue to practice. Because once expertise is gone, we won’t miss it until the day we need it most.