Exploring the Development of Bayesian Statistics and Statistical Philosophy

4min Snip

00:00

Play full episode

Summary

Transcript

Episode notes

Reverend Thomas Bayes, a non-conformist minister in the mid-18th century, revolutionized statistical philosophy by highlighting that probabilities are not facts about the world but statements of personal ignorance. This insight challenges the rigid objectivity of science, emphasizing the subjective nature of statistical estimation. It introduced a fundamental split in statistics between frequentists and Bayesians, where Bayesians focus on determining how likely a hypothesis is to be true given experimental results, incorporating personal judgment into statistical analysis.

At its simplest, Bayes’s theorem describes the probability of an event, based on prior knowledge of conditions that might be related to the event. But in Everything Is Predictable, Tom Chivers lays out how it affects every aspect of our lives. He explains why highly accurate screening tests can lead to false positives and how a failure to account for it in court has put innocent people in jail. A cornerstone of rational thought, many argue that Bayes’s theorem is a description of almost everything.

But who was the man who lent his name to this theorem? How did an 18th-century Presbyterian minister and amateur mathematician uncover a theorem that would affect fields as diverse as medicine, law, and artificial intelligence? Fusing biography and intellectual history, Everything Is Predictable is an entertaining tour of Bayes’s theorem and its impact on modern life, showing how a single compelling idea can have far reaching consequences.

Tom Chivers is an author and the award-winning science writer for Semafor. Previously he was the science editor at UnHerd.com and BuzzFeed UK. His writing has appeared in The Times (London), The Guardian, New Scientist, Wired, CNN, and more. He was awarded the Royal Statistical Society’s “Statistical Excellence in Journalism” awards in 2018 and 2020, and was declared the science writer of the year by the Association of British Science Writers in 2021. His books include The Rationalist’s Guide to the Galaxy: Superintelligent AI and the Geeks Who Are Trying to Save Humanity’s Future, and How to Read Numbers: A Guide to Stats in the News (and Knowing When to Trust Them). His new book is Everything Is Predictable: How Bayesian Statistics Explain Our World.

Shermer and Chivers discuss: Thomas Bayes, his equation, and the problem it solves • Bayesian decision theory vs. statistical decision theory • Popperian falsification vs. Bayesian estimation • Sagan’s ECREE principle • Bayesian epistemology and family resemblance • paradox of the heap • Reality as controlled hallucination • human irrationality • superforecasting • mystical experiences and religious truths • Replication Crisis in science • Statistical Detection Theory and Signal Detection Theory • Medical diagnosis problem and why most people get it wrong.