#94 Psychometrics Models & Choosing Priors, with Jonathan Templin
Learning Bayesian Statistics
Navigating Model Comparison in Bayesian Psychometrics
This chapter explores the intricacies of model comparison within the Bayesian framework, advocating for the use of multiple evaluation metrics. It highlights the significance of posterior predictive model checking and addresses the challenges of discrete data in psychometrics, while discussing best practices in both traditional and modern approaches.
Proudly sponsored by PyMC Labs, the Bayesian Consultancy. Book a call, or get in touch!
In this episode, Jonathan Templin, Professor of Psychological and Quantitative Foundations at the University of Iowa, shares insights into his journey in the world of psychometrics.
Jonathan’s research focuses on diagnostic classification models — psychometric models that seek to provide multiple reliable scores from educational and psychological assessments. He also studies Bayesian statistics, as applied in psychometrics, broadly. So, naturally, we discuss the significance of psychometrics in psychological sciences, and how Bayesian methods are helpful in this field.
We also talk about challenges in choosing appropriate prior distributions, best practices for model comparison, and how you can use the Multivariate Normal distribution to infer the correlations between the predictors of your linear regressions.
This is a deep-reaching conversation that concludes with the future of Bayesian statistics in psychological, educational, and social sciences — hope you’ll enjoy it!
Our theme music is « Good Bayesian », by Baba Brinkman (feat MC Lars and Mega Ran). Check out his awesome work at https://bababrinkman.com/ !
Thank you to my Patrons for making this episode possible!
Yusuke Saito, Avi Bryant, Ero Carrera, Giuliano Cruz, Tim Gasser, James Wade, Tradd Salvo, William Benton, James Ahloy, Robin Taylor,, Chad Scherrer, Zwelithini Tunyiswa, Bertrand Wilden, James Thompson, Stephen Oates, Gian Luca Di Tanna, Jack Wells, Matthew Maldonado, Ian Costley, Ally Salim, Larry Gill, Ian Moran, Paul Oreto, Colin Caprani, Colin Carroll, Nathaniel Burbank, Michael Osthege, Rémi Louf, Clive Edelsten, Henri Wallen, Hugo Botha, Vinh Nguyen, Marcin Elantkowski, Adam C. Smith, Will Kurt, Andrew Moskowitz, Hector Munoz, Marco Gorelli, Simon Kessell, Bradley Rode, Patrick Kelley, Rick Anderson, Casper de Bruin, Philippe Labonde, Michael Hankin, Cameron Smith, Tomáš Frýda, Ryan Wesslen, Andreas Netti, Riley King, Yoshiyuki Hamajima, Sven De Maeyer, Michael DeCrescenzo, Fergal M, Mason Yahr, Naoya Kanai, Steven Rowland, Aubrey Clayton, Jeannine Sue, Omri Har Shemesh, Scott Anthony Robson, Robert Yolken, Or Duek, Pavel Dusek, Paul Cox, Andreas Kröpelin, Raphaël R, Nicolas Rode, Gabriel Stechschulte, Arkady, Kurt TeKolste, Gergely Juhasz, Marcus Nölke, Maggi Mackintosh, Grant Pezzolesi, Avram Aelony, Joshua Meehl, Javier Sabio, Kristian Higgins, Alex Jones, Gregorio Aguilar, Matt Rosinski, Bart Trudeau, Luis Fonseca and Dante Gates.
Visit https://www.patreon.com/learnbayesstats to unlock exclusive Bayesian swag ;)
Links from the show:
- Jonathan’s website: https://jonathantemplin.com/
- Jonathan on Twitter: https://twitter.com/DrJTemplin
- Jonathan on Linkedin: https://www.linkedin.com/in/jonathan-templin-0239b07/
- Jonathan on GitHub: https://github.com/jonathantemplin
- Jonathan on Google Scholar: https://scholar.google.com/citations?user=veeVxxMAAAAJ&hl=en&authuser=1
- Jonathan on Youtube: https://www.youtube.com/channel/UC6WctsOhVfGW1D9NZUH1xFg
- Jonathan’s book: https://jonathantemplin.com/diagnostic-measurement-theory-methods-applications/
- Jonathan’s teaching: https://jonathantemplin.com/teaching/
- Vehtari et al. (2016), Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC: https://arxiv.org/abs/1507.04544
- arviz.plot_compare: https://python.arviz.org/en/stable/api/generated/arviz.plot_compare.html
- LBS #35, The Past, Present & Future of BRMS, with Paul Bürkner: https://learnbayesstats.com/episode/35-past-present-future-brms-paul-burkner/
- LBS #40, Bayesian Stats for the Speech & Language Sciences, with Allison Hilger and Timo Roettger: https://learnbayesstats.com/episode/40-bayesian-stats-speech-language-sciences-allison-hilger-timo-roettger/
- Bayesian Model-Building Interface in Python: https://bambinos.github.io/bambi/
Abstract
You have probably unknowingly already been exposed to this episode’s topic - psychometric testing - when taking a test at school or university. Our guest, Professor Jonathan Templin, tries to increase the meaningfulness of these tests by improving the underlying psychometric models, the bayesian way of course!
Jonathan explains that it is not easy to judge the ability of a student based on exams since they have errors and are only a snapshot. Bayesian statistics helps by naturally propagating this uncertainty to the results.
In the field of psychometric testing, Marginal Maximum Likelihood is commonly used. This approach quickly becomes unfeasible though when trying to marginalise over multidimensional test scores. Luckily, Bayesian probabilistic sampling does not suffer from this.
A further reason to prefer Bayesian statistics is that it provides a lot of information in the posterior. Imagine taking a test that tells you what profession you should pursue at the end of high school. The field with the best fit is of course interesting, but the second best fit may be as well. The posterior distribution can provide this kind of information.
After becoming convinced that Bayes is the right choice for psychometrics, we also talk about practical challenges like choosing a prior for the covariance in a multivariate normal distribution, model selection procedures and more.
In the end we learn about a great Bayesian holiday destination, so make sure to listen till the end!
Transcript
This is an automatic transcript and may therefore contain errors. Please get in touch if you're willing to correct them.
Remember Everything You Learn from Podcasts
