🔒Making sure AI works: UMass Chan’s new AI Assurance Lab tests developing healthcare tech, so it functions properly in the real world

A decade after the beginning of the AI boom of the 2010s, artificial intelligence has not only been integrated in our homes, our jobs, and our phones, but our health care.

That can bring about a lot of concern, said Dr. David McManus, chair and professor of medicine at UMass Chan Medical School in Worcester. AI can now step in to assist with diagnosing, predicting health outcomes, hospital triage, and more. These tools must be void of biases and discriminatory practices, McManus said. Enter UMass Chan’s AI Assurance Lab, a novel initiative established in April to test the ethical use of AI in health care. By deploying and studying the impacts of AI tools across different healthcare providers and patient subgroups, the AI Assurance lab is working to add the human back into artificial intelligence. “We wanted to ensure that in trying to do something good, we didn't exacerbate health disparities further by training a model that might, without realizing it, be prejudiced in some way,” said McManus. McManus and Dr. Adrian Zai, UMass Chan’s chief research informatics officer, co-lead the Assurance Lab, where healthcare businesses test their AI tools for real-world efficacy and equity. “Even if the overall accuracy appears strong, it doesn't mean that [the AI] is performing in a fair approach,” said Zai. “Ethical concern is usually created by unequal impact, not just statistical differences.”

Stress testing AI

“One of the biggest misconceptions about healthcare AI is that good performance of a model throughout the development process does … guarantee good performance in the real world,” said Zai.

The lab works with companies to containerize their AI, bringing the new technology into the lab’s testing facility. Through analyzing AI tools, the Assurance Lab looks for safety, reliability, and the potential for errors causing patient harm. The lab conducts stress testing, evaluating performance and capabilities under hypothetical, extreme conditions. “We evaluate whether clinicians actually understand and appropriately contextualize the outputs,” Zai said. If the results produced by an AI tool aren’t transparent or obvious to providers, that creates an ethical risk, especially when the AI is used to inform care decision making, he said. The lab often uses real-word data from the UMass Memorial Health system hospitals in Central Massachusetts. In one case, Zai and McManus built a database with millions of digitized EKG results from racially diverse patients being treated at UMass, to test the performance of an AI tool designed to predict cardiovascular complications in pregnant women.

Unintended consequences

The AI Assurance Lab runs tests through UMass Chan’s interprofessional Center for Experiential Learning and Simulation lab. iCELS is a center focused on simulating real-life scenarios to train students, professionals, and technologies. AI is impacting how providers deliver care, how patients receive care, and how both interact, said Dr. Melissa Fischer, iCELS executive director.

“Anytime you make a change in a complex system like a healthcare system, it's best to study it from multiple perspectives … and say ‘What might the intended and unintended outcomes be?’” Fischer said. iCELS is testing a new AI app designed to assess whether less invasive technology can be used in predictive testing. The lab has used simulations with volunteers to analyze how potential patients interact with the technology, how providers interact with the information, and if the AI gives providers enough information to make clinical decisions. Volunteers are walked through the application to see how easily they can access, download, understand, and use the technology. iCELS only test AI tools on volunteers, and not real patients, though Fischer hopes to one day have patients interact with providers around AI. “How do you communicate about it in a patient care experience, and how does that impact the clinician patient experience? How does that impact any potential trust between the clinician and the patient?” she said.

Unsexy AI

When McManus and Zai originally started the Assurance Lab, they thought they’d focus on AI in clinical tools to help with diagnoses and to better prepare lesser-trained providers. As the lab rounds out its first ninth months of operation, that target has shifted.

“Health AI applications should be scoped to include things that are really not that sexy,” said McManus. Those unsexy applications include acting as a scribe during appointments or more administrative tasks, like deciding the order in which a fleet of vehicles should be sent out or scheduling patients for colonoscopies. “A lot of where I see health AI going is to start first, potentially, with getting people comfortable with AI, with some of the things that aren't as risky or as hard or as threatening,” McManus said. McManus reads news articles saying AI will replace doctors, but he said that concept isn’t going to pass any legal muster anytime soon. In a life-or-death industry like health care, patients won’t jump to be the first to have AI oversee their care. “Today, you need to have a human process involved. No AI directly makes any decision without human oversight,” said Zai. “It's going to be like that for a little while until there's a much better safety blanket.” Mica Kanner-Mascolo is a staff writer at Worcester Business Journal, who primarily covers the healthcare, manufacturing, and higher education industries.