A new machine learning model developed by Mount Sinai researchers could reduce overcrowding in emergency departments, improve staff efficiencies and boost patient outcomes by informing ED decision-making and streamlining patient throughput, a new study shows.
WHY IT MATTERS
Admission decisions and bed placements often result in ED care delays that can compromise patient outcomes and have been statistically tied to increased mortality rates, according to researchers from the Mount Sinai Health System's Department of Emergency Medicine and its Icahn School of Medicine.
"Emergency surgery patients with delays had a 53% increase in mortality, and each delayed patient had an increased length of stay of 2.6 days and an increased cost of $3,335," said researchers in a new report published in July's Mayo Clinic Proceedings: Digital Health.
Previous studies have shown that when nurses expressed certainty in their predictions of patient admissions, they outperformed other AI scoring tools, suggesting that nurse insights could enhance tools and better enable earlier decision-making and more efficient bed allocations.
By comparing an ML approach with nurse predictions for hospital admissions across the diverse Mount Sinai health system, they aimed to demonstrate a prediction model with high enough accuracy to be used to augment bed planning.
The researchers compared a machine learning model trained on structured and routine triage data for more than 1 million emergency room visits with 574 real nurses' triage assessments to determine if their predictions would enhance the model's predictive capabilities.
"Machine learning-based predictions outperformed triage nurse estimates for hospital admissions," they concluded.
For the study, "Comparing Machine Learning and Nurse Predictions for Hospital Admissions in a Multisite Emergency Care System," the researchers used the Bio-Clinical BERT LLM and the XGBoost algorithm in real-time hospital workflows, and then asked triage nurses at six hospitals to make an admissions prediction for their adult patients.
The participating EDs have a combined annual volume of about 500,000 patients, according to the report.
Researchers said that they used triage data from 1.8 million ED visits that occurred between January 2019 and December 2023 to train the model. It was then tested on 46,912 prospective ED visits with recorded nurse predictions from Sept. 1 to Oct. 31, 2024.
Triage nurses were blinded to the model's predictions.
The model had an accuracy of 85.4% and a sensitivity of 70.8% at a 0.30 probability threshold, according to the report. Nurses' predictions had an 81.6% accuracy rate and 64.8% sensitivity rate, according to the report.
They also tested whether combining nurse predictions with the ML output would improve predictive accuracy beyond the model alone. It did not, the researchers said.
The incorporating nurses' predictions achieved an accuracy of 86.2%. In contrast, the ML model alone (at a 0.5 probability threshold) achieved an accuracy of 86.7%. Thus, adding nurses' predictions did not improve overall accuracy.
This result suggests that nurse-based predictions did not provide incremental value when integrated with the model's output.
The researchers said that additional modeling of novel workflows using the AI and human-in-the-loop predictive admissions data would be the next step to determining if there is a reduction in ED length of stay before they could pilot a model in a live clinical environment. They said they may also evaluate if this tool is helpful to inpatient teams. If inpatient teams can anticipate their needs, they can "start increasing staffing prior to a predicted admission peak," the researchers noted.
THE LARGER TREND
The Froedtert and the Medical College of Wisconsin health network optimized bed capacity and coordination between departments with AI, ML and data analytics approaches.
"Seasonal surges, unplanned admissions and fluctuating patient needs make it challenging to maintain an optimal allocation of resources," Ravi Teja Karri, an ML engineer at Froedtert Health, told Healthcare IT News in December.
A proactive approach that created anticipatory workflows rather than reactive ones enabled "more accurate forecasting of patient volumes and better interdepartmental coordination," he said.
By integrating the forecasting, the health system turned "predictions into action for hospital staff, which resulted in enhanced patient care and overall efficiency."
ON THE RECORD
"Although the human-in-the-loop concept did not increase the accuracy of predicting admissions beyond what is currently possible with only the nurse or the AI model, when we added the human prediction to the AI model, we found that the results remained fairly constant," Mount Sinai researchers said in the new report.
"The admission prediction information could be used to initiate steps in the admission process hours earlier than would otherwise be possible," they added. "The admission prediction information could be used to initiate steps in the admission process hours earlier than would otherwise be possible since we are not waiting for a provider to finish evaluating the cases; theoretically creating a reservation list to allow for bed planning in parallel to the initial ED workup and stabilization."
Andrea Fox is senior editor of Healthcare IT News.
Email: afox@himss.org
Healthcare IT News is a HIMSS Media publication.
