This is a republication of the paper “The Epic Sepsis Model Falls Short — The Importance of External Validation” with the title above. The article is preceded by the Key Messages by the Editor of the Site.
KEY MESSAGES [1]
- In healthcare, a model is typically trained for a single purpose like sepsis prediction and distributed as install-anywhere software.
- These models are trained to perform a classification or prediction task using a mix of biological inputs (such as laboratory test results, which are more stable across patient populations) and operational inputs (such as care patterns, which are variable and tend to be hospital-specific).
- But models often have poor generalization, which in turn limits their use.
- Epic, a top EHR vendor, recently began retraining their sepsis model on a hospital’s local data before deployment after the algorithm was widely criticized for poor performance.
- The need to retrain every model for the specific patient population and hospital where it will be used creates cost, complexity, and personnel barriers to using AI.
- This is where foundation models can provide a mechanism for rapidly and inexpensively adapting models for local use.
- Rather than specializing in a single task, foundation models capture a wide breadth of knowledge from unlabeled data.
- Then, instead of training models from scratch, practitioners can adapt an existing foundation model, a process that requires substantially less labeled training data.
For example, the current generation of medical foundation models have reported reducing training data requirements by 10x when adapting to a new task.
For clinical natural language extraction tasks, variations of large foundation models like GPT-3 can achieve strong performance using only a single training example.
Notes [1]
[1] Based on: “How Foundation Models Can Advance AI in Healthcare”, published by “HAI Stanford”, by “Jason Fries, Ethan Steinberg, Scott Fleming, Michael Wornow, Yizhe Xu, Keith Morse, Dev Dash, Nigam Shah”, on Dec 15, 2022
ORIGINAL PUBLICATION
The Epic Sepsis Model Falls Short — The Importance of External Validation
The ESM did not identify 67% of patients with sepsis,
despite generating alerts for an ESM score of 6 or higher for 18% of hospitalized patients, thus creating a large burden of alert fatigue.
JAMA Network, Editorial
Anand R. Habib, MD, MPhil1,2; Anthony L. Lin, MD1; Richard W. Grant, MD, MPH3,4
June 21, 2021
statsnews
Sepsis accounts for nearly 1 million hospitalizations annually and is a major contributor to hospital length of stay, health care expenditures, and in-hospital mortality (ranging from 12.5%-15%).1
Early sepsis identification allows care teams to promptly implement goal-directed therapy to mitigate clinical deterioration.
In this issue of JAMA Internal Medicine, Wong et al2 report on their external validation of the Epic Sepsis Model (ESM), a prediction tool available within the Epic electronic health record that is designed to generate automated alerts that warn clinicians that patients may be developing sepsis.
Based on their examination of 38 455 hospitalizations at the University of Michigan (Ann Arbor) between December 2018 and October 2019, Wong et al2 found that the ESM had a sensitivity of 33%, specificity of 83%, positive predictive value of 12%, and negative predictive value of 95%, with an area under the curve of 0.63 (95% CI, 0.62–0.64).
This falls short of the area under the curve of 0.76 to 0.83 that was jointly reported by Epic and University of Colorado Health.3
Despite generating alerts on 18% of all patients, the ESM did not detect sepsis in 67% of patients with sepsis.
Originally published https://jamanetwork.com
About the authors
Anand R. Habib, MD, MPhil1,2;
Anthony L. Lin, MD1;
Richard W. Grant, MD, MPH3,4
1 Department of Medicine,
University of California, San Francisco
2 Editorial Fellow,
JAMA Internal Medicine
3 Division of Research,
Kaiser Permanente Northern California, Oakland
4 Associate Editor,
JAMA Internal Medicine
ORIGINAL PUBLICATION
External Validation of a Widely Implemented Proprietary Sepsis Prediction Model in Hospitalized Patients
JAMA Network
Andrew Wong, MD1; Erkin Otles, MEng2,3; John P. Donnelly, PhD4; et alAndrew Krumm, PhD4; Jeffrey McCullough, PhD5; Olivia DeTroyer-Cooley, BSE6; Justin Pestrue, MEcon6; Marie Phillips, BA7; Judy Konye, MSN, RN8; Carleen Penoza, MHSA, RN8; Muhammad Ghous, MBBS4; Karandeep Singh, MD, MMSc1,4
June 21, 2021
KEY POINTS
Question
How accurately does the Epic Sepsis Model, a proprietary sepsis prediction model implemented at hundreds of US hospitals, predict the onset of sepsis?
Findings
In this cohort study of 27 697 patients undergoing 38 455 hospitalizations, sepsis occurred in 7% of the hosptalizations. The Epic Sepsis Model predicted the onset of sepsis with an area under the curve of 0.63, which is substantially worse than the performance reported by its developer.
Meaning
This study suggests that the Epic Sepsis Model poorly predicts sepsis; its widespread adoption despite poor performance raises fundamental concerns about sepsis management on a national level.
ABSTRACT
Importance
The Epic Sepsis Model (ESM), a proprietary sepsis prediction model, is implemented at hundreds of US hospitals. The ESM’s ability to identify patients with sepsis has not been adequately evaluated despite widespread use.
The Epic Sepsis Model (ESM), a proprietary sepsis prediction model, is implemented at hundreds of US hospitals. The ESM’s ability to identify patients with sepsis has not been adequately evaluated despite widespread use.
Objective
To externally validate the ESM in the prediction of sepsis and evaluate its potential clinical value compared with usual care.
Design, Setting, and Participants
This retrospective cohort study was conducted among 27 697 patients aged 18 years or older admitted to Michigan Medicine, the academic health system of the University of Michigan, Ann Arbor, with 38 455 hospitalizations between December 6, 2018, and October 20, 2019.
Exposure
The ESM score, calculated every 15 minutes.
Main Outcomes and Measures
Sepsis, as defined by a composite of (1) the Centers for Disease Control and Prevention surveillance criteria and (2) International Statistical Classification of Diseases and Related Health Problems, Tenth Revision diagnostic codes accompanied by 2 systemic inflammatory response syndrome criteria and 1 organ dysfunction criterion within 6 hours of one another.
Model discrimination was assessed using the area under the receiver operating characteristic curve at the hospitalization level and with prediction horizons of 4, 8, 12, and 24 hours.
Model calibration was evaluated with calibration plots. The potential clinical benefit associated with the ESM was assessed by evaluating the added benefit of the ESM score compared with contemporary clinical practice (based on timely administration of antibiotics).
Alert fatigue was evaluated by comparing the clinical value of different alerting strategies.
Results
We identified 27 697 patients who had 38 455 hospitalizations (21 904 women [57%]; median age, 56 years [interquartile range, 35–69 years]) meeting inclusion criteria, of whom sepsis occurred in 2552 (7%).
The ESM had a hospitalization-level area under the receiver operating characteristic curve of 0.63 (95% CI, 0.62–0.64).
The ESM identified 183 of 2552 patients with sepsis (7%) who did not receive timely administration of antibiotics, highlighting the low sensitivity of the ESM in comparison with contemporary clinical practice.
The ESM also did not identify 1709 patients with sepsis (67%) despite generating alerts for an ESM score of 6 or higher for 6971 of all 38 455 hospitalized patients (18%), thus creating a large burden of alert fatigue.
The ESM also did not identify 1709 patients with sepsis (67%) despite generating alerts for an ESM score of 6 or higher for 6971 of all 38 455 hospitalized patients (18%), thus creating a large burden of alert fatigue.
Conclusions and Relevance
This external validation cohort study suggests that the ESM has poor discrimination and calibration in predicting the onset of sepsis.
The widespread adoption of the ESM despite its poor performance raises fundamental concerns about sepsis management on a national level.
This external validation cohort study suggests that the ESM has poor discrimination and calibration in predicting the onset of sepsis.
The widespread adoption of the ESM despite its poor performance raises fundamental concerns about sepsis management on a national level.
About the authors
Andrew Wong, MD1; Erkin Otles, MEng2,3; John P. Donnelly, PhD4; Andrew Krumm, PhD4; Jeffrey McCullough, PhD5; Olivia DeTroyer-Cooley, BSE6; Justin Pestrue, MEcon6; Marie Phillips, BA7; Judy Konye, MSN, RN8; Carleen Penoza, MHSA, RN8; Muhammad Ghous, MBBS4; Karandeep Singh, MD, MMSc1,4
Author Affiliations
1Department of Internal Medicine,
University of Michigan Medical School, Ann Arbor
2Medical Scientist Training Program, University of Michigan Medical School, Ann Arbor
3Department of Industrial and Operations Engineering, University of Michigan College of Engineering, Ann Arbor
4Department of Learning Health Sciences, University of Michigan Medical School, Ann Arbor
5School of Public Health, University of Michigan, Ann Arbor
6Department of Quality,
Michigan Medicine, Ann Arbor
7Health Information Technology and Services, Michigan Medicine, Ann Arbor
8Nursing Informatics, Michigan Medicine, Ann Arbor
