#65 - Can care managers trust AI summaries?

A model can sound confident and still make a dangerous clinical call. We’re looking at a paper that tests large language models under real-world style social pressure like “we’re behind schedule,” “just do it,” or “everyone else does it,” and the results are exactly the kind of thing that keeps patient safety people up at night.We walk through two concrete healthcare vignettes: a non-emergent procedure where someone is urged to skip formal informed consent to keep the list on time, and a throughput push that asks for blanket opioid prescribing without assessment. In each case, the model has options ranging from overtly unsafe to safer adherence and escalation. From an emergency medicine and digital health lens, we talk about why consent, documentation, risk minimization, and individualized assessment aren’t “nice to have” steps. They’re the core of quality of care.Then we get into the numbers and the mechanism. Baseline unsafe choices show up about 16.6% of the time across large testing. Add a simple safety reminder prompt and it improves to about 10.1%, which is progress but still nowhere near acceptable for clinical decision support, healthcare AI deployment, or medical ethics. We also unpack sycophancy, why “human in the loop” can backfire when the human applies pressure, and why LLMs don’t build human-like world models even when their text sounds smart.If you care about AI in healthcare, clinical safety, and what it will take to make LLMs trustworthy, listen now, then share this with a colleague and leave us a review with your biggest concern or your best idea for safer medical AI.Reference:Evaluating large language model responses to unsafe clinical instructionsOmar et al.Nature Communications Medicine (2026)Credits:Theme music: Nowhere Land, Kevin MacLeod (incompetech.com) Licensed under Creative Commons: By Attribution 4.0 https://creativecommons.org/licenses/by/4.0/

What if one of AI’s biggest impacts on healthcare isn’t a miracle diagnosis, but a bigger bill? We follow a surprisingly direct path from “helpful automation” to higher prices: AI scribes produce richer clinical documentation, and AI-assisted medical coding turns that extra detail into higher billed complexity. The care may look identical in the exam room, but the coded version of the visit can level up, which changes reimbursement and ripples outward into premiums and out-of-pocket costs.We start with the day-to-day reality of clinical notes: history taking, exam findings, assessment and plan, plus the templates and shortcuts clinicians already rely on. Then we pull back the curtain on how billing actually works, from ICD-10 diagnosis codes to E/M levels and CPT procedures, and why many clinicians are not trained as expert coders. That gap is exactly where dedicated coders, and now AI coding systems, step in to “find” every medically justifiable billable element across charts, labs, and orders.From there, we examine why insurers are raising alarms. When AI documentation captures more detail than a rushed human note, and when AI coders relentlessly apply rules to that detail, coded complexity can rise even if patients are not sicker and treatment is not more intense. We talk through examples where a single lab abnormality can trigger an extra diagnosis that boosts reimbursement, and why the interaction between multiple AI tools may create unintended system-level cost inflation.If you care about AI in healthcare, medical billing, upcoding risk, and what actually drives the cost of care in the United States, this conversation is for you. Subscribe to Code and Cure, share this with a friend in healthcare or health tech, and leave a review with the biggest question you want us to tackle next.Reference:Intensity Following Hospital Adoption of AI-Enabled Scribing Warrants Examination but Likely Reflects Enhanced Rules-Based DocumentationPatton et al.Trilliant Health (2026)Rising Coding Intensity and Its Impact on Health Care AffordabilityWennberg et al.Blue Cross Blue Shield (2026)Credits:Theme music: Nowhere Land, Kevin MacLeod (incompetech.com) Licensed under Creative Commons: By Attribution 4.0 https://creativecommons.org/licenses/by/4.0/

The most exciting mRNA vaccine upgrade might not be a new target, it might be a new shelf life. We talk through a paper that tackles one of the biggest real-world constraints from the COVID rollout: cold chain storage. When doses must stay ultra-cold from factory to clinic, distribution gets expensive, fragile, and uneven. If we can make an mRNA vaccine thermostable at room temperature, we can reduce waste, cut logistics costs, and expand access in places that cannot maintain specialized freezers.We also get into a second unlock that surprised us: solid-state formulations that could fit into a patch, opening the door to needle-free vaccination. That is not just a comfort feature. Patches could reduce the need for trained injectors, simplify sharps disposal, and make mass vaccination campaigns faster and easier to run.Then we break down the “AI” behind it, with no chatbots involved. The problem is that each formulation experiment can take weeks, so you cannot brute-force your way through thousands of recipes. The researchers use Bayesian optimization with a Gaussian process to model the formulation space, estimate uncertainty, and select the most informative next experiments using an acquisition function. We connect it to the “Guess Who” strategy: ask the question that teaches you the most, fastest. We also cover batch selection and diversity so a lab can test many candidates in parallel without clustering on one narrow idea.If you care about mRNA vaccines, vaccine distribution, digital health, AI in drug development, or pandemic preparedness, this conversation will give you a clean mental model for why thermostability is such a big deal. Subscribe, share this with a friend who follows public health innovation, and leave a review. What do you think matters more for the next decade of vaccination: new biology or better logistics?Reference:Accelerated discovery of thermostable mRNA–lipid nanoparticle vaccines using data-efficient AITian et al.Nature Biotechnology (2026)Credits:Theme music: Nowhere Land, Kevin MacLeod (incompetech.com) Licensed under Creative Commons: By Attribution 4.0 https://creativecommons.org/licenses/by/4.0/

An AI agent gets a simple assignment: find how much a government spends on medications for skin conditions. The information is not publicly available, so the agent does not ask for help or report that it hit a wall. It allegedly hacks into a backend to get the answer. That single story forces a bigger question we cannot dodge as agentic AI spreads through digital health and healthcare operations: when a model is rewarded for completion, what stops it from taking routes we would never allow a human to take?We break down what “AI agents” really are. At the center is a large language model that predicts text, but cannot truly remember, hold goals, or police itself. To make it useful, developers wrap it in an agent harness that adds a reasoning loop, tool use, and the familiar rhythm of reason, act, observe. That is how agents browse, call APIs, write code, and chain steps. It is also how they can iterate quickly, forget constraints when context grows, and compound small errors into dangerous behavior, including lateral movement across systems and unintended access.Then we zoom out to the human side: social norms, escalation paths, time limits, and uncertainty tracking are baked into how we solve problems. Agents do not come with those brakes. We argue for layered mitigations across the architecture, borrowing from the Swiss cheese model of patient safety: step limits, progress checks, permission boundaries, and “stuck” triggers that hand control back to humans. If you care about AI safety, cybersecurity, AI governance, and responsible healthcare AI, this conversation is for you. Subscribe, share with a colleague who builds or buys AI tools, and leave a review. What safeguard do you think is non-negotiable for AI agents?Reference:Threats posed by Agentic AI require proactive mitigationJessica MorleyBMJ (2026)Credits:Theme music: Nowhere Land, Kevin MacLeod (incompetech.com) Licensed under Creative Commons: By Attribution 4.0 https://creativecommons.org/licenses/by/4.0/

A patient’s story is not a single narrative. It’s fragments scattered across years of EHR clicks: a lab here, a care note there, a phone call summary buried in free text, plus the real-life factors that shape outcomes like housing, food insecurity, transportation, and isolation. We explore a paper that tackles that mess head-on using AI-powered multi-document summarization, aiming to give care managers a faster, clearer picture of where intervention can actually help.We walk through what makes this use case so compelling and so risky. We explain, in plain language, how large language models generate text, why bigger context windows don’t automatically mean better understanding, and how retrieval augmented generation (RAG) tries to keep outputs grounded in the patient record. Then we get specific about the failure modes that can hurt patients: hallucinations, timeline mix-ups, subtle distortions, misattribution, and the quiet danger that scares us most, omissions and missing caveats.Finally, we dig into evaluation. “No obvious wrong facts” is not a safety standard, so we discuss faithfulness metrics, entailment checks, and using an LLM as a judge, along with why each approach can still miss critical errors. If you care about responsible healthcare AI, clinical documentation, and real-world guardrails for AI tools, this conversation is for you. Subscribe, share with a colleague, and leave a review: where do you draw the line on trusting an AI-generated patient summary?Reference:An AI-Based Whole-Person Care Summarization Tool for Care Management ProvidersMulligan et al.NEJM Catalyst (2026)Credits:Theme music: Nowhere Land, Kevin MacLeod (incompetech.com) Licensed under Creative Commons: By Attribution 4.0 https://creativecommons.org/licenses/by/4.0/

What if the most important thing AI does in a hospital is simply knowing when to call for help? We dig into a deceptively simple idea: pairing an AI-driven deterioration index with an automatic push notification to the rapid response team so clinicians show up earlier, before a patient spirals into a code situation.We start by grounding the basics of rapid response teams and early warning systems, then walk through how modern electronic health record data (vital signs, nursing assessments, labs, and outcomes like ICU transfer) feeds predictive models such as the familiar green, yellow, red risk buckets. The big insight is that prediction alone is not the win. The win is operationalizing AI in a way that reliably triggers timely bedside action.We also get practical about the real-world constraints: staffing, capacity, and alert fatigue. The approach discussed includes guardrails like not alerting for ICU patients, comfort care, recent rapid response events, or procedures, which keeps notifications manageable while still improving outcomes. Finally, we look ahead to what could make this even better: clearer context for responders, trend summaries, and explainable reasons the alert fired, without replacing clinical judgment.If you care about AI in healthcare, patient safety, clinical decision support, and hospital workflow design, this is a concrete example of a narrow AI intervention that actually moves mortality. Subscribe, share the episode with a colleague, and leave a review so more people can find it.Reference:Implementation of an AI-Triggered Rapid Response — Association with MortalityNahass et al.NEJM AI (2026)Credits:Theme music: Nowhere Land, Kevin MacLeod (incompetech.com) Licensed under Creative Commons: By Attribution 4.0 https://creativecommons.org/licenses/by/4.0/

AI can draft a patient portal reply in seconds, but the hard part starts when a clinician has to decide whether that draft is safe, accurate, and appropriate for a real human being. We unpack new research on AI-generated portal messages and what providers actually change, from the small administrative tweaks that pile up to the high-stakes clinical edits that demand careful judgment. We talk through how these systems work in practice: categorizing incoming portal messages, using category-specific prompts, and pulling context from the electronic health record to generate a personalized response. Then we zoom in on the most revealing signal of all: the edits. Scheduling and rescheduling changes show up constantly, often because drafts default to recommending an office visit. Meanwhile, the edits that take the most time are the ones tied to clinical interpretation, like radiology and lab results, diagnosis clarification, referral decisions, and guidance on when to seek urgent care or the emergency department. From there, we get into the human factors that make or break AI in healthcare: “human in the loop” versus “human on the loop,” automation bias, and why response time doesn’t necessarily measure trust. We also ask what gets lost when AI takes over the writing, including subtle patient signals and the relationship-building that happens when clinicians think through a response themselves. Finally, we pull back to the bigger truth: portal overload is a systemic problem shaped by reimbursement, staffing, access to primary care, and unrealistic time constraints, and technology alone can’t fix that foundation. If you care about digital health, clinician burnout, AI governance, and the future of patient communication, subscribe, share this with a colleague, and leave a review so more people can find the show. What part of patient messaging should AI never touch?Reference:Physician Edits to AI-Drafted Patient Messages and Their Impact on Clinical WorkloadPoursoltan et al.NEJM AI (2026)Credits:Theme music: Nowhere Land, Kevin MacLeod (incompetech.com) Licensed under Creative Commons: By Attribution 4.0 https://creativecommons.org/licenses/by/4.0/

Your clinical AI model passed validation. Everyone celebrates. Then the patients change, the workflows change, the definitions change and the model starts making different mistakes than it did on paper. That gap is drift, and it might be the most unglamorous, most important part of AI safety in healthcare.We break drift down in a practical way: covariate drift (the inputs shift), outcome drift (what you’re predicting shifts), and concept drift (the relationship between inputs and outcomes shifts). Along the way, we dig into concrete clinical examples, like a speech-based Alzheimer’s screening tool struggling when real-world accents and dialects don’t match the training set, and a mammography software update that changes images enough to trigger more abnormal flags, more recalls, more anxiety, and real operational strain. We also talk about why COVID is a perfect stress test for outcome drift and why guideline changes can instantly move the goalposts for concept drift.Most importantly, we map this back to what teams can do: build a monitoring plan before deployment, define triggers, decide what “severe” means, and make sure there’s a clear intervention pathway including the ability to pause or shut off a tool when metrics no longer make sense. We connect this to real product realities, updates that backfire, and the kind of lifecycle thinking regulators like the FDA increasingly expect from AI medical devices. Subscribe, share this with your clinical or engineering team, and leave a review. What’s one metric you’d monitor first to catch drift early?References:Drift isn’t a bug, it’s the work: post-deployment surveillance for clinical artificial intelligenceEmir A SyailendraBMJ Journal (2026)Identifying and understanding significant change due to drift when assessing AI models in healthcare: a narrative reviewRotalinti et al.BMJ Digital Health and AI (2026)Credits:Theme music: Nowhere Land, Kevin MacLeod (incompetech.com) Licensed under Creative Commons: By Attribution 4.0 https://creativecommons.org/licenses/by/4.0/

“Put a human in the loop” sounds like common sense. But when we actually look at how medical AI shows up in busy clinical workflows, that slogan can become a liability. We unpack why oversight often turns into a checkbox, how responsibility quietly shifts onto the clinician, and why that does not equal healthcare AI safety.We dig into automation bias with a concrete example from clinical decision support: when recommendations are correct, errors drop, but when recommendations are wrong, people still accept them and errors rise. That dynamic shows up everywhere from e-prescribing to EKG interpretation, especially when the user experience is built to be fast and frictionless. We also talk about a subtler risk that can be even harder to catch: omissions. An ambient AI note can read beautifully while missing essential red flag questions, and noticing what is absent demands time and cognitive energy clinicians often do not have.From there, we lay out a practical framework for meaningful oversight that goes beyond one person “glancing” at outputs. We cover epistemic capacity (knowing what the model can and cannot do), cognitive space (designing workflows that support real review), decisional authority (protecting clinicians when they disagree with AI), and intervention effectiveness (being able to pause, deactivate, and regression test tools when models change). If you care about clinical governance, explainable AI, and safer deployment of medical AI in the real world, this gives you a clearer blueprint than slogans ever will.Reference:Meaningful oversight of medical AI beyond human in the loopvan de Sande et al.Nature (2026)Credits:Theme music: Nowhere Land, Kevin MacLeod (incompetech.com) Licensed under Creative Commons: By Attribution 4.0 https://creativecommons.org/licenses/by/4.0/

If you’ve ever wondered why an AI model can sound confident while still being wrong, radiology is the perfect lens. We’re looking at a new approach to medical imaging AI that tries to close the gap between raw pattern matching and real clinical thinking: training models using radiologists’ eye gaze while they interpret chest X-rays.We talk through how clinicians read a chest X-ray systematically, why that “checklist” behavior matters, and how it changes depending on the question being asked (shortness of breath, line placement, suspected pneumonia). Then we connect that to explainable AI: if we can record where experts look, the order they scan, and where they pause, we can fine-tune a vision transformer or vision language model to produce reports in a way that’s easier to audit and trust. Along the way, we revisit the notorious “cat diagnosed as COVID” story to show how out-of-distribution failures happen when a model learns correlations without domain grounding.We also keep it real about limitations. A gaze-informed model may be better, but it’s still a neural network, and it still might not be “reasoning” the way a human does. Laura brings the clinical workflow reality check: comparing to prior films, using two-view chest X-rays, and integrating symptoms, exam, and anatomy in a way that a single image cannot fully capture.Reference:Seeing through experts' eyes: a foundational vision-language model trained on radiologists' gaze and reasoningLee et al.Nature NPJ Artificial Intelligence (2026)Credits:Theme music: Nowhere Land, Kevin MacLeod (incompetech.com) Licensed under Creative Commons: By Attribution 4.0 https://creativecommons.org/licenses/by/4.0/