AI systems have to be pressure-tested in real-world clinical settings to ensure correct reasoning, accuracy and reliability.
getty
Although it has always been well known that AI systems will get facts wrong occasionally in daily use, a new emerging problem is increasingly worrying clinical and technology leaders: AI convincingly appearing competent when it is completely incorrect and does not possess any real competence in an area at all. This phenomenon has been termed “cognitive spoofing” by industry experts, which refers to AI’s ability to confidently project expertise when it actually lacks the clinical context or judgement required to make a decision, especially in matters of care delivery. Because AI has the ability to create responses that are well polished and may seem backed by evidence, clinical leaders and physicians who put too much confidence into these systems can face mishaps in their daily workflows.
A study by Microsoft found that the latest frontier models were able to score exceedingly high on medical benchmarks and examinations. However, upon stress testing, the study found that these systems leveraged smart answering techniques rather than actual knowledge or sound reasoning: “Leading systems often guess correctly even when key inputs like images are removed, flip answers under trivial prompt changes, and fabricate convincing yet flawed reasoning. These aren’t glitches; they expose how today’s benchmarks reward test-taking tricks over medical understanding.” This means that although these systems score high on objective examinations, they may by no means be ready for actual clinical settings or stresses. As the study describes, clinical benchmarks for AI evaluation often emphasize correctness rather than reasoning or how that correct answer was reached. However, if this approach is pressure-tested in real time settings, it could lead to significant issues: “Medical readiness is a multidimensional construct. In real-world settings, models must tolerate missing or noisy data, justify their decisions in a manner clinicians can understand, and reason across time, modality, and context. Performance must be not only accurate but also reliable, interpretable, and safe under uncertainty.”
Therein lies a deeper problem. How do end users and physicians known when they are being spoofed, or if the answer is perhaps incorrect but the AI has convinced them otherwise?
One key answer is the concept of explainaibility, which is a broader term that refers to getting AI systems to justify or clarify how they derived their outputs. This is important as it helps users understand exactly how the model worked through reasoning to get to its final answer. As explained by IBM, “It is crucial for an organization to have a full understanding of the AI decision-making processes with model monitoring and accountability of AI and not to trust them blindly…Explainable AI also helps promote end user trust, model auditability and productive use of AI. It also mitigates compliance, legal, security and reputational risks of production AI.”
So what do users have to do, especially in clinical settings?
Challenge the systems constantly. When provided with an output, query how the AI got there, ask for its sources, and prompt the system to transparently show its reasoning. Then, double check all of the sources and actually click on the links to make sure the outputs match the source. Yes, it will take additional time; however, it is certainly worth the extra time if it I can mitigate an error, especially in a clinical setting.