SGEM#361: Under My Umbrella, Ella, Ella – Review of Meta-Analyses in Emergency Medicine

Date: February 24th, 2022 Reference: Parish et al. An umbrella review of effect size, bias, and power across meta-analyses in emergency medicine. AEM 2021 Guest Skeptic: Professor Daniel Fatovich is an emergency physician and clinical researcher based at Royal Perth Hospital, Western Australia. He is Head of the Centre for Clinical Research in Emergency Medicine, Harry Perkins Institute of Medical Research; Professor of Emergency Medicine, University of Western Australia; and Director of Research for East Metropolitan Health Service. Case: A resident has been following the literature over their four years of training. They have already seen several things come into fashion and go out of fashion during this short time. This includes therapeutic hypothermia for out-of-hospital cardiac arrest (OHCA), tranexamic acid (TXA) for epistaxis and electrolyte solutions for mild pediatric gastroenteritis. They wonder how strong the evidence is for much of what we do in emergency medicine. Background: There are many things in medicine that could be considered myth or dogma. We have covered some of these over the 10 years. Topical anesthetic uses of 24-48 hours for mild cornea abrasions will cause blindness- No (SGEM# 315) Epinephrine for adult out-of-hospital cardiac arrests (OHCAs) results in better neurologic outcomes – No (SGEM#238) TXA for intracranial hemorrhage, isolated traumatic brain injury, post-partum hemorrhage or gastrointestinal bleed results in better primary outcomes - No (SGEM#236, SGEM#270, SGEM#214, and SGEM#301) Therapeutic hypothermia in adult OHCA saves lives – No (SGEM#336) Electrolyte solutions are needed in mild pediatric gastroenteritis - No (SGEM#158) A lot of medical practice is based on low quality research. Tricoci et al. JAMA Feb 2009 looked at the ACC/AHA guidelines from 1984 to 2008. They found 53 guidelines with 7,196 recommendations. The results were only 11% of recommendations were considered Level A, 39% were Level B and 50% were Level C. The definitions used for each level of evidence are as follows: An update was published by Fanaroff et al in JAMA 2019. The level of high-quality evidence had not changed much when looking at the ACC/AHA guidelines from 2008-2018. There were 26 guidelines with 2,930 recommendations. Now Level A recommendations were down to 9%, Level B 50% and Level C 41%. This lack of evidence is not isolated to cardiology. A recent study looked at the top ten elective orthopaedic procedures. It was an umbrella review of meta-analyses of randomized control trials (RTCs) or other study designs if no RCTs existed (Blom et al BMJ 2021). The comparison was the clinical efficacy of the most common orthopaedic procedures with no treatment, placebo, or non-operative care. The primary outcome was the quality of the evidence for each procedure. Only two out of ten common procedures, carpal tunnel decompression and total knee replacement, showed superiority over non-operative care. Clinical Question: What is the effect of faults such as underpowered studies, flawed studies (i.e. methodologic and statistical errors, poorly designed studies) and biases in the field of therapeutic interventions in the emergency medicine literature? Reference: Parish et al. An umbrella review of effect size, bias, and power across meta-analyses in emergency medicine. AEM 2021 Population: SRMAs 1990-2020 in the top 20 journals under the google scholar subcategory: emergency medicine; emergency medicine meta-analyses from JAMA, NEJM, BMJ, The Lancet, and the Cochrane Database of Systematic Reviews; emergency medicine topics across all PubMed journals; and an extraction of all studies from the Annals of Emergency Medicine Systematic Review Snapshots (SRS) series. Exclusions: Articles were excluded if they did not include a quantitative synthesis (meta-analysis); did not contain at least two summarized studies; did not make a comparison between two groups to assess an effect size; did not report an effect size as at least one of mean difference or standardized mean difference (SMD; Cohen's d), odds ratio (OR), risk ratio (RR), hazard ratio (HR), or transformations of these effect sizes; were meta-analyses of diagnostic accuracy studies; or were not related to the practice of emergency medicine. Intervention: Data supplement 1 lists all 431 MAs derived from 332 published SRMAs. Comparison: Includes placebo, usual care, nothing. Outcomes: Identify broad patterns in study parameters (effect size, power, mortality benefit and potential bias). Dr. Austin Parish We are fortunate to have the lead author on this episode even though it is not an SGEMHOP. Dr. Austin Parish is the Chief Resident in Emergency Medicine at the Lincoln Medical Center, Bronx NY. He is also a researcher for the Meta Research Innovation Center at Stanford (METRICS) Authors’ Conclusions: “Few interventions studied within SRMAs relevant to emergency medicine seem to have strong and unbiased evidence for improving outcomes. The field would benefit from more optimally powered trials.” Quality Checklist for Therapeutic Systematic Reviews: The clinical question is sensible and answerable. Yes The search for studies was detailed and exhaustive. Yes The primary studies were of high methodological quality. No The assessment of studies were reproducible. Yes The outcomes were clinically relevant. Yes There was low statistical heterogeneity for the primary outcomes. No The treatment effect was large enough and precise enough to be clinically significant. Sometimes Results: The systematic review identified 431 eligible meta-analyses (MAs) relevant to emergency medicine. The MAs included a total of 3,129 individual study outcomes of which 2,593 (83%) were from randomized controlled trials. Key Result: A minority of interventions published in SRMA and relevant to emergency medicine have unbiased and strong evidence for improved outcomes. Primary Outcome: Broad patterns in study parameters Effect Size: The median Odds Ratio (OR) across all studies was 0.70. Within each MA, the earliest study effect on average demonstrated larger benefit compared to the overall summary effect. Only 57 of 431 meta-analyses (13%) both favored the experimental intervention and did not show any signal of small study effects or excess significance. Power: Only 12 of 431 MAs had at least one study with 80% or higher power to detect an OR of 0.70 Mortality: Zero out of 431 MAs reported the interventions significantly decreased mortality in well-powered trials. Although the power of studies increased somewhat over time, most studies were underpowered. Biases: 92 of the SRMAs included 10 or more studies that could be analyzed with a funnel plot for asymmetry. 25% (23/92) showed evidence of asymmetry suggesting excess significance. 85 (20%) of the SRMAs reported statistical significance in favor of the intervention. Of these, 1/3 showed a signal of small study effect and/or excess significance while 2/3 (57/85) did not. Of the 57, only 36 (63%) had a GRADE assessment reported. Half were rated as low-quality evidence and only 11% rated as high-quality evidence. 1. How Good is the Evidence? I’ve often posed the question: what proportion of our EM clinical practice is backed up by high level evidence? After speaking with thought leaders the answer I got to was less than 10%. This umbrella review quantifies the answer in more detail: 12/431 = 2.8%. There is not a large amount of high-level evidence supporting most EM practices. The results demonstrate that very few interventions meet the highest evidence standards, and most of the SRMAs are significantly flawed and may overstate true treatment effects. So, we need to advance our knowledge and practice through never ending questioning of it, via a research culture, whereby clinical trials and clinical research are a routine part of everyday EM work, research that engages clinicians and patients with clinically useful questions – to be a learning health system. What is the proportion of our EM clinical practice is backed up by high level evidence? 2. The Best Evidence: Table 1 in the paper lists the 12 MAs in EM that have statistically significant results (p < 0.05 by random effects), based on data with no signal for small study effects or excess significance and at least one RCT and at least one study with 80% power to detect a small effect (d = 0.2). The biggest effect of an intervention was the rate of haemolysis using straight needle venepuncture vs an IV; OR 0.11(95%CI; 0.05-0.23). Of the 12 MAs, only another three had a 95% confidence intervals that did not cross 1 (the line of no statistical difference), for well powered studies (fixed effect): senior doctor vs no senior doctor in triage for preventing patient left without being seen (OR 0.74, 95% CI; 0.70-0.77); clopidogrel pre-treatment vs no clopidogrel pre-treatment in acute coronary syndrome patients to receive percutaneous intervention (OR 0.79, 95% CI; 0.73-0.85) for a major coronary event; glucocorticoids and usual care vs usual care for croup (OR 0.44, 95% CI; 0.27-0.72) on rate of return visits. While there were no mortality benefits listed under fixed effect, well powered studies, under the heading of random effects, all studies – there were some mortality benefits for mechanical CPR, transfer for angioplasty, thrombolysis for PE and vasopressin + catecholamines. The details will be listed in the blog. Mechanical CPR vs manual CPR for OHCA on mortality by arrival to hospital (OR 0.80, 95% CI; 0.68-0.94); Transfer for angioplasty vs on site thrombolysis for ST elevated myocardial infarction on 30-day mortality (OR 0.78, 95% CI; 0.61-0.99); Thrombolysis vs conventional anticoagulation for pulmonary embolism (OR 0.42, 95% CI; 0.19-0.93);