AI Health Check: No Governance, No Trust

The enormous potential of AI in healthcare is matched only by its very real challenges.

One proprietary AI tool designed as an early warning system for sepsis, for example, couldn’t differentiate high- and low-risk patients before they received treatments. The system’s findings, it turns out, were no better than a coin flip.

Is this just one instance of technology not living up to its promise? Maybe. But it also highlights a broader risk. Trust is paramount when it comes to AI in healthcare. If doctors and patients don’t feel they can trust the information an AI tool gives them, they won’t use the tool.

And that lack of adoption has bottom-line consequences. Hospital systems and other providers are pouring massive amounts of time and money into developing AI solutions. When those investments don’t pan out, they’re not just minor line-item corrections – they can derail an entire digital strategy. 

That’s why building and maintaining trust has to be one of the primary objectives of any healthcare AI project. Just as you wouldn’t ignore or downplay the key technical aspects of a new AI tool or solution, you can’t take shortcuts when it comes to governance.

Here’s how to keep trust at the forefront of your development process.

The core risks of AI in healthcare (and why trust breaks fast)

The risks associated with AI are not unique to healthcare projects. But because sensitive patient data and care plans are involved, the loss of trust that accompanies those risks can be much more damaging than in other industries.

AI hallucinations, for example, can be a problem in everything from academic research to financial transactions. Nobody likes inaccurate information. But the consequences of an AI hallucination producing an incorrect diagnosis, triggering a false alert, or giving a patient the wrong medical recommendation can be much more serious.

Hallucinations are an output issue. But there’s also risk on the input side of the AI equation, specifically in terms of data drift as those models are being trained:

• What are the sources of the training data, and how is the accuracy and reliability of that data validated?
• Does it accurately reflect the patient population that will be affected by the tool?
• Have regional variations and contextual differences been accounted for?

When AI hallucinations, bias, and data drifts show up in clinical settings, trust erodes quickly – often after just one or two visible mistakes. For example, if a patient discovers errors in the AI-generated notes coming from their doctor, they may decide not to follow future instructions.

That’s more than an AI failure. Effective care is built on the trust inherent in the doctor-patient relationship. If that trust starts to wane, quality treatment becomes impossible.

Building trust means focusing on AI governance

Healthcare, of course, is a famously regulated industry. Dealing with the multitude of inconsistent, incompatible, and overlapping rules for patient data may, in fact, be one of the biggest challenges facing hospital systems and other healthcare providers as they undertake digital transformation projects.

But the risks surrounding data only increase when AI enters the mix. And with AI regulatory frameworks still evolving – and not always quickly enough to keep up with the technology they’re meant to oversee – providers need to take it upon themselves to build trust into their AI infrastructures.

That’s why AI development should be a direct outcome of a well-designed governance framework. At a minimum, providers should focus on establishing a strong governance committee that recognizes its vital role in building AI. This committee should be…

• Active. The committee should look at the results of AI experimentation over time and monitor closely for hallucinations and data inconsistencies.
• Watchful. The committee should be mindful of automation bias by reviewing workflows to ensure that the organization still has ample human input in the development process and is not becoming overly reliant on AI.
• Proactive. By establishing guardrails and policies before a new AI tool is rolled out rather than only after the fact in response to an incident, the committee can build trust into the process.
• Ongoing. Governance only works when it is cyclical and continuous – it should never be a static piece of the puzzle.

Ideally, this committee should be run by one person who is dedicated solely to the responsibility of governance. In many organizations, this has led to the establishment of a chief AI officer role, a position tasked not with bringing new AI tools and solutions to the table, but rather the ongoing oversight and maintenance of those tools as they start to come online.

The real-world barriers facing AI governance 

Of course, if it was easy to set up an effective governance committee, every organization would do it. But like most aspects of AI development, governance too has its share of challenges, including:

• Decentralized or unstructured workflows that make it hard to apply a unified standard of governance across the organization.
• The speed of AI adoption and development, which in many cases has outpaced oversight.
• A fragmented approach to AI among different groups within the organization, which can lead to a shadow IT problem.

Shadow IT, in particular, highlights a distinct difference between traditional healthcare tech transformations and AI.

In the past, if a hospital system needed to upgrade its digital imaging capabilities, for example, it was a major organization-wide project. Big, expensive machines needed to be purchased and installed, and everyone was trained on how to use them.

But with AI tools, the barrier of entry is often much lower.

If a handful of doctors don’t like working with the AI scribe tool the hospital has rolled out, they can simply opt for an alternative solution that works better for them. And if this happens in multiple groups across the organization, effective monitoring and governance becomes next to impossible.

Give AI a better chance to meet its potential

The company that rolled out that AI sepsis tool had the right idea in mind. If doctors knew in advance which of their patients were more at risk for developing sepsis, they could take the appropriate steps to address those conditions before they became a bigger problem. In other words, AI could help them improve patient care.

But it didn’t work out that way. Development got out ahead of governance, meaning issues that could have been uncovered and addressed weren’t. And the tool didn’t meet its potential.

Unfortunately, those kinds of misses affect future behavior. Will doctors or patients who are burned by one AI solution trust the next one they’re given? Probably not. That’s why every provider rolling out AI tools has to understand this risk and build governance into its development process.

They may not get a second chance to earn that trust.

Photo: Thanakorn Lappattaranan, Getty Images

This post appears through the MedCity Influencers program. Anyone can publish their perspective on business and innovation in healthcare on MedCity News through MedCity Influencers. Click here to find out how.