Since becoming first in nation to implement new emergency room AI program, UMass Memorial has seen accuracy improve
Photo | Timothy Doyle
UMass Memorial Medical Center - University Campus
As the use of artificial intelligence is becoming more prevalent in health care, UMass Memorial Medical Center’s emergency room in Worcester last year quietly became the first level-one trauma center in the United States to implement KATE AI.
Now, after nearly 18 months, UMass Memorial has seen a significant increase in the accuracy of its initial assessment of the severity of patients’ conditions.
KATE AI, an artificial intelligence emergency room triage support system which helps correctly determine the severity of a patient’s condition, has been used in UMass Memorial’s emergency room since February 2023, helping nurses catch errors, streamlining triaging, and decreasing the number of patients leaving the ER without being seen.
During triage, a process in which the nurses assess the patient’s condition, the nurse will assign the patient with an emergency severity index (ESI) level from 1 (most urgent) to 5 (least urgent). While triage is taking place, KATE reads the patient's medical records, scanning hundreds of data points throughout them in order to determine an appropriate ESI level. Only if the nurse’s chosen ESI level differs from KATE’s will an alert appear, informing the nurse of medical information that a patient did not or could not disclose during triage.
Ken Shanahan, senior director of emergency medicine and behavioral health at UMass Memorial, gave the example that patients may come into the ER with a fever, but not disclose they’re also immunocompromised, leading the nurse to assign the patient a lower ESI. In an instance like this, KATE will have scanned the patient’s records and pinpoint that they’re immunocompromised. If KATE determines a different ESI level than the nurse, the AI will bring this information to the nurse’s attention.
KATE’s alerts help to flag errors made when inputting data during the triage process. If a nurse accidentally swaps a patient’s heart and respiratory rates, KATE can alert a patient with a heart rate of 18 should be assigned a higher ESI level, allowing the nurse to recognize the error and adjust the record.
Before UMass Memorial deployed Kate, the hospital’s ESI accuracy rate was 55%, which is 5 percentage points lower than the national average of approximately 60%, a statistic reported in a study published by the Journal of Emergency Nursing. Since the hospital implemented KATE and updated its ESI, UMass Memorial’s ESI accuracy rate has risen by 10 percentage points to 65%.
Shanahan originally started shopping around for AI support due to what he refers to as the perfect storm he walked into when joining UMass Memorial in 2020. The COVID pandemic coupled with the mental health crisis meant the ER was already seeing more patients. Then the nurses of Saint Vincent Hospital in Worcester went on strike, limiting Saint Vincent’s capacity and flooding more patients towards UMass Memorial. At the same time, the hospital had implemented its Hospital at Home program, meaning more nurses were out of the ER and attending to patients off site. The hospital was experiencing a high nurse turnover as nurses left to be travel nurses and new travel nurses came in.
“We just knew we needed to do something different, and our waiting room was our biggest risk,” Shanahan said. “So we wanted to make sure we're adequately assessing that risk and making sure we're bringing the patients back that are the sickest first. And so we needed some help with that.”
Even with the mounting struggles facing UMass Memorial, the hospital’s nurses expressed trepidation about KATE’s implementation, fearing the technology would be used to penalize those who made a mistake. It took a concerted effort on Shanahan’s part to assuage those fears.
“It was really talking through the union and having that conversation and our staff and say ‘That's not what the purpose or intent of this is. It's not to get anyone in trouble. It's to have your back,’” said Shanahan.
Shanahan had purposefully picked an AI system that would not interfere with the nurses’ regular triage process. He wanted the technology to aid their usual practice, not change it. Determining an ESI level is always up to the nurse, even if KATE disagrees, he said.
Though KATE has information the nurses may not be privy to during triage, Shanahan said nurses have contextual information KATE doesn’t. For example, a patient’s heart rate may be elevated, something KATE would flag for the nurse if the nurse assigned them a lower ESI level, but the patient might have advised the elevated heart rate is due to their anxiety in hospitals, leading the nurse to remain confident in the original ESI determination.
While KATE doesn't have contextual information, it can help eliminate bias from triage assessments. For example, Shanahan said pre-KATE nurses would sometimes triage based on what was going on in the department and capacity levels as opposed to solely what was going on medically with the patient. UMass Memorial’s ER has a rapid flow area (RFA) for patients with less acute ESI levels to be seen and discharged within a couple of hours. Pre-KATE, Shanahan said some nurses would assign a lower ESI even if a patient had more acute symptoms in order to get them into the RFA and seen by a provider faster.
Though done with good intentions, incorrectly triaging someone would clog up the RFA with patients needing several hours of care. Having KATE alert a patient needs to be triaged at a higher level eliminates bias. The hospital is now no longer seeing those errors made, Shanahan said.
In fact, KATE flagged a serious error on the first day it was deployed, a correction Shanahan said made the whole year-long process to implement the technology worth it.
Though he wouldn’t divulge specifics, Shanahan said prior to using KATE, the hospital had performed a root cause analysis and implemented a corrective action plan in response to errors made in a specific case surrounding properly assigning an ESI level. The first day KATE was implemented, the same complaint came in and the same error was made, but KATE caught it.
“Everything worked. It was like, right there, that shows the benefit of the product. And it was literally on day one,” Shanahan said.
Mica Kanner-Mascolo is a staff writer at Worcester Business Journal, who primarily covers the healthcare and diversity, equity, and inclusion industries.
WBJ Web Partners
Most Recent
Most Recent
0 Comments