A Christmas Debate: Statistically Speaking

Date: December 15, 2024 Guest Skeptics: Dr. Chris Carpenter, Vice Chair of Emergency Medicine at Mayo Clinic. Today, we’re sleighing through the holiday season with a special episode filled with statistical cheer, a dash of skepticism, and a hint of eggnog-flavoured nerdiness. This is an SGEM Xtra like the one we did on What I Learned from Top Gun. It’s fun to mix it up and not do a structured critical appraisal of a recent publication and have a more philosophical chat.  The inspiration for the SGEM Xtra episode is the BMJ 2022 holiday article called "The 12 Days of Christmas, the Statistician Gave to Me,". The BMJ statistical editors gifted us a publication highlighting common statistical faux pas, laid out in true holiday spirit. We came up with our own SGEM 12 nerdy days. The 12 Days of Christmas the SGEM Gave to Me Day 1: A P Value in a Pear Tree Ah, the P value of frequentists statistics —often misunderstood and frequently abused. The key here is remembering that a P value isn’t proof of the “truth”. We define truth as the best point estimate of an observed effect size with a confidence interval around the point estimate. We should not hang our clinical decisions on a single P value. When do we ever base our care on one data point? Day 2: Two Confidence Intervals Confidence intervals (CIs) tell us the range of plausible values for our estimate. If they’re too wide, it’s like a holiday sweater that’s too loose—unflattering and not very useful. Reconsider that sample size or effect size to get clinically impactful intervals you would want to share with Santa Claus. And remember, CIs don’t mean certainty! Day 3: Three Missing Values Missing data is the Grinch of research. Ignoring it or using improper methods to handle it can bias your results in unpredictable directions. Using methods like multiple imputation or sensitivity analysis can salvage your data without sacrificing rigour. We often forget that missing data extends beyond interventional studies. For example, in diagnostic research like emergency ultrasound indeterminate results are often swept under the rug and either excluded or not reported – even when methods like 3x2 tables exist to improve the transparency of these indeterminate results that reflect real-life when trying to deploy these diagnostics that require technical expertise and subjective interpretation. Day 4: Four Overfit Models Overfitting is like over-decorating your Christmas tree—too many ornaments and it collapses. Overfit models describe the noise, not the signal, making them poor predictors in new datasets. Keeping your models simple and robust can pay downstream dividends for widespread usability and replicability. Day 5: Five Golden Rules Here are the five statistical golden rules for a randomized control trial: Power Calculation: Do an a priori power calculation that defines the expected effect size, choose a significance level (α, commonly 0.05 and two-sided), specify the desired power (commonly 80-90%) and account for anticipated dropouts. Randomization: Use proper randomization techniques (simple, stratified, or block randomization). Ensure allocation concealment to prevent prediction of group assignment. Outcomes: Clearly defined primary and secondary endpoints. Pick patient-oriented outcomes (POO) rather than disease-oriented outcomes (DOOs), surrogate-oriented outcomes (SOOs) or monitor-oriented outcomes (MOOs) Statistical Analysis Plan: Provides a roadmap for how data will be analyzed and reduces the risk of data dredging or p-hacking. Specify which tests will be used, pre-plan subgroup and sensitivity analyses and define how missing data will be handled. Control of Bias and Confounding: Bias and confounding can distort study results and lead to incorrect conclusions. Use blinding (single, double, or triple) to reduce performance and detection biases. Collect baseline characteristics to assess the balance between gro...