Patients across the world are increasingly seeking medical guidance from artificial intelligence systems. In a recent Times of India report, after multiple specialists failed to identify the cause of a woman’s chronic cough and internal bleeding, her daughter turned to ChatGPT. The system asked a question clinicians had not: whether the patient was taking a blood pressure medication known to cause such symptoms. When the medication was changed, her condition improved.
When empathy and fluency shape trust
Episodes like this are becoming more common as AI tools enter everyday healthcare, in part because they are perceived as more attentive: responding immediately, asking follow-up questions without visible impatience, and expressing conclusions with fluency and confidence.
A 2023 study published in JAMA Internal Medicine found that patients rated AI-generated medical responses as significantly more empathetic and trustworthy than physicians’ replies. For individuals navigating rushed visits and fragmented care, AI interactions can feel emotionally safer than clinical encounters.
But fluency is not the same as expertise. Under stress, patients may mistake polished tone for accuracy. AI is fast enough to shortcut reflection and fluent enough to discourage questioning. When it fills gaps in access, time, and empathy, it risks becoming not just a source of information, but a substitute for clinical judgment.
Clinicians are not immune to similar dynamics. Adoption of AI tools is accelerating across healthcare. OpenEvidence, a medical AI company, estimates that more than 100 million Americans will be treated this year by physicians using AI-supported tools. Ambient documentation systems, a commonly used tool, reduce administrative burden and free physicians to focus on patients. Yet early evidence suggests that AI scribes can flatten nuance, failing to capture distress or psychosocial context. In high-pressure environments, peer-reviewed studies show that radiologists may defer to AI triage labels more than intended, a form of automation bias that can contribute to delayed care.
These failures are rarely about negligence. They emerge when tools are introduced into workflows without clear boundaries around responsibility. Missing guardrails allow mistakes to scale quickly.
Individual vigilance is not enough. As AI becomes more embedded in care, the cues that shape trust and authority are shifting faster than our oversight structures can adapt. Performing well in controlled settings is only part of the picture, the harder question is whether AI is being integrated into clinical workflows in ways that keep responsibility, authority, and accountability properly aligned.
Designing for judgment, not just accuracy
One emerging framework for addressing these challenges is computational humility. This approach designs systems that foreground uncertainty, make model limitations visible, and preserve human judgment rather than obscure it. That means considering whether an AI is accurate, and deciding when its output should be questioned, overridden, or ignored. That requires attending to technical performance, the emotional vulnerabilities of patients, the professional autonomy of clinicians and the real-world context in which decisions are made.
It also means aligning financial incentives with patient-centered outcomes. Value-based payment models such as Medicare’s Shared Savings and Value-Based Purchasing programs reward health systems for reducing readmissions and improving chronic disease outcomes rather than increasing service volume. In such environments, AI tools are evaluated not simply on technical performance, but on whether they improve real-world outcomes, workflow usability, and patient safety. When reimbursement and accountability are shared across developers, providers, and payers, the incentive shifts from deploying tools quickly to integrating them responsibly.
AI is already embedded in healthcare. What remains unsettled is the relationship modern medicine is willing to build with it: one grounded in fluency and convenience, or one structured around clarity, limits, and shared accountability.
Photo: Irina_Strelnikova, Getty Images
Leslie Pascaud is a strategic insights and marketing leader with over 35 years of experience driving growth across B2B, B2C, and nonprofit organizations in health tech and global health. She specializes in translating complex innovation into clear, differentiated narratives that resonate with both commercial and mission-driven audiences. As Chief Marketing Officer of a digital clinical trials company, she led cross-functional teams to develop thought leadership and go-to-market strategies that contributed to three consecutive years on the Inc. 5000 list of fastest-growing companies.
She is a Strategic Advisor at Kinetic Strategic Consulting Group and a three-term board advisor to Tiko, a 2025 recipient of the Audacious Project grant for its “big bold solution” via a pioneering digital platform that provides access to health services for young African women. Leslie’s current work focuses on the evolving role of artificial intelligence in healthcare, particularly how AI systems influence clinical judgment, patient trust, and the alignment of responsibility in care delivery.
This post appears through the MedCity Influencers program. Anyone can publish their perspective on business and innovation in healthcare on MedCity News through MedCity Influencers. Click here to find out how.
