SGEM#448: More than A Feeling – Gestalt vs CDT for Predicting Sepsis

Reference: Knack et al. Early Physician Gestalt Versus Usual Screening Tools for the Prediction of Sepsis in Critically Ill Emergency Patients. Ann Emerg Med 2024 Date: July 25, 2024 Guest Skeptic: Dr. Justin Morgenstern is an emergency physician and the creator of the #FOAMed project called www.First10EM.com Case: Your hospital is running Morbidity and Mortality (M&M) rounds after a few recent cases in which the diagnosis of sepsis was identified late, and antibiotics were delayed. They are planning on instituting a mandatory screening tool at triage, and one of the main purposes of the meeting is to compare the available tools, such as qSOFA and MEWS. As the local evidence-based medicine (EBM) nerd, they ask for your opinion on the evidence. Background: Sepsis is a life-threatening organ dysfunction caused by a dysregulated host response to infection. It is a medical emergency that requires prompt recognition and treatment to improve patient outcomes.  We have covered the topic of sepsis many times on the SGEM (SGEM#69, SGEM#90, SGEM#92, SGEM#113, SGEM#168, SGEM#174, SGEM Xtra, SGEM#346, SGEM#371 and SGEM Peds Xtra). There is a lot of emphasis on identifying sepsis early, with the idea that early intervention will save lives. However, despite a strong push for early antibiotics, the evidence of benefit is mostly lacking. There is observational data that is widely cited to suggest that early completion of sepsis bundles improves outcomes but considering that physicians don’t purposefully delay antibiotics in patients with known sepsis, this data is severely limited by multiple confounders [1]. A randomized control trial (RCT) done in the prehospital setting enrolled 2,698 patients. They were randomized to ceftriaxone 2gm intravenous (IV) in the ambulance or usual cares (fluids and supplementary oxygen) until arrive to the ED. The primary outcome reported was no statistical difference in mortality at 28 days (8% in both groups) despite giving antibiotics 96 minutes earlier [2]. All of the secondary outcomes (mortality at 90 days, misdiagnoses, hospital length of stay, ICU admission rate, ICU length of stay, and quality of life) also did not show a statistical difference between the intervention group and the control group (SGEM#207). Thus, early identification of sepsis might not be as important as sometimes stated in guidelines. However, getting the right diagnosis is clearly important for our patients, and as good as we are, no clinician is perfect. Acknowledging our imperfections, many have suggested that screening tools or decision tools might help increase accuracy when screening for sepsis. Many such tools exist, such as the Systemic Inflammatory Response Syndrome (SIRS), Sequential Organ Failure Assessment (SOFA), quick SOFA (qSOFA) and theModified Early Warning (MEWS) score. Unfortunately, the enthusiasm for decision instruments often outstrips the evidence. For a decision instrument to benefit patients, it needs to have more than a high sensitivity. It needs to change physician practice for the better. It needs to be better than current clinical practice, or just plain clinical judgement. There is an article published in AEM, with an author list that includes the who’s who of decision rules – from Jeff Kline to Nathan Kupperman to my BFF Chris Carpenter. That document tells us “Before widespread implementation, CDRs should be compared to clinical judgement.” [3] Unfortunately, most of our rules haven’t cleared this basic evidentiary hurdle. Dave Schriger and colleagues reviewed publications in the Annals of Emergency Medicine from 1998 to 2015 and found that only 11% of the studies compared decision aids to clinical judgement [4]. In those that did compare to clinical judgement, physician judgement was superior in 29% and equivalent or mixed in 46%. The decision aid only outperformed clinical judgement in 10% of papers (or two total trials). A similar review by Sanders et al 2015 concludes that clinical decision rules “are rarely superior to clinical judgement and there is generally a trade-off between the proportion classified as not having disease and the proportion of missed diagnoses.” [5] Therefore, before widespread use of sepsis tools like qSOFA or the MEWS score, we really need to see comparisons to clinical judgment. That brings us to the current study, which aims at comparing a number of these tools to initial clinical judgment in the emergency department. Clinical Question: What is the accuracy of standardized screening tools and a machine learning model to predict a hospital discharge diagnosis of sepsis, compared with physician gestalt in the hyperacute period immediately after patient presentation among undifferentiated patients with critical illness in the emergency department (ED)? Reference:  Knack et al. Early Physician Gestalt Versus Usual Screening Tools for the Prediction of Sepsis in Critically Ill Emergency Patients. Ann Emerg Med 2024 Population: Critically ill, adult (18 and older), undifferentiated medical patients presenting to the specialized four bed resuscitation area in this emergency department. Excluded: Patients with trauma, and obvious causes of illness, defined as cardiac arrest, STEMI, suspected stroke, and patients in active labour. They also excluded patients being transferred from outside facilities. Intervention: Faculty emergency physicians were asked “what is the likelihood that this patient has sepsis?”and asked to rate the likelihood on a scale from 0 to 100. They were asked 15 and 60 minutes after the patient’s presentation. To calculate statistics, they decided anything above 50% was consistent with the diagnosis of sepsis, but it is not clear that is a good assumption, which we will discuss below. Comparison: The physician gestalt was compared to SIRS, SOFA, qSOFA, MEWS, and a logistic regression machine learning model using Least Absolute Shrinkage and Selection Operator (LASSO) for variable selection. Outcome: A final diagnosis of sepsis, based on ICD 10 codes at discharge. Type of Study: This is a single center prospective observational trial. Authors’ Conclusions: “Among adults presenting to an ED with an undifferentiated critical illness, physician gestalt in the first 15 minutes of the encounter outperformed other screening methods in identifying sepsis”. Quality Checklist for Observational Study: Did the study address a clearly focused issue? Yes Did the authors use an appropriate method to answer their question? Yes Was the cohort recruited in an acceptable way? Yes Was the exposure accurately measured to minimize bias? No Was the outcome accurately measured to minimize bias? No Have the authors identified all-important confounding factors? Unsure Was the follow up of subjects complete enough? Yes How precise are the results? The confidence intervals are tight enough to believe Do you believe the results? Yes Can the results be applied to the local population? Unsure Do the results of this study fit with other available evidence? Yes Funding of the Study: The authors did not report any specific funding sources and declared no conflicts of interest. Results: They included 2,484 patients, with a median age of 53, 60% being male and 11% (257/2,484) were ultimately diagnosed with sepsis. Most physician judgment (94%) was completed by physicians with only 6% being completed by residents. They were missing a lot of data for the other screening tools. The median visual analog scale (VAS) score in patients with sepsis was 81, as compared to 8 in those without sepsis. Key Result: Physician gestalt was better than all the decision tools, both at 15 and 60 minutes. Primary Outcome: Sepsis diagnosis 1. Lack of Gold Standard: What is the true definition of sepsis, and do we even have a gold standard? In this study, the (fool’s) gold standard was the chart containing an ICD 10 code of sepsis at the time of discharge. But how many of these patients truly had sepsis? More importantly, not all sepsis is created equal. I might care a lot about identifying septic shock or severe sepsis, but if these patients fell out of those more severe categories, do we even care? Finally, discharge diagnosis it a poor gold standard, because it is possible that patients could have developed sepsis later in their hospital stay. Imagine a patient with intestinal ischemic as the cause of their initial presentation. Even if that patient later develops sepsis, we have done the patient no good by labelling them sepsis in the ED and missing their dying intestines. In fact, they provide us with a table of the 10 patients who were ultimately diagnosed with sepsis but “missed” by the initial clinician. The exact case I invented, a patient with intestinal ischemia and zero SIRS criteria, is represented as a miss. Perhaps most importantly, antibiotics were given in the ED to every single ‘miss’, which really makes you wonder about the definition of ‘missed sepsis’ being used. 2. VAS Score: They asked physicians to rate the chances of sepsis from 0 to 100. That is a reasonable question for research purposes, but it is entirely unclear what these numbers mean for clinical care. If a patient has a 60% chance of sepsis, do you empirically treat as sepsis, or wait for more information? 40%? 20%? It is likely that different clinicians will act at different thresholds. For their stats, they decided that anything above 50% meant the patient had sepsis, but they didn’t ask the clinicians for their interpretation. Would the treating physicians have agreed? It is possible that they were giving empiric antibiotics to patients with even a 20% chance of sepsis, which would make the 50% cutoff meaningless. Therefore, although this is a theoretically interesting question,