John Hattie is an education researcher from New Zealand with a very ambitious goal: to synthesise the myriad quantitative research studies on education in a single publication. The number of articles affecting his book Visible Learning numbers in the region of 80 thousand (!). The results of his analysis have been hailed as the "Holy Grail" of education by such prestigious authorities as the Times Education Supplement. So, how did he and his team do it?

Hattie uses an approach known as meta-analysis. Meta-analyses take numerous research articles trying to measure an effect and compare them in order to ultimately determine the size of the effect. They are common in medicine, where they are often used to elucidate whether a drug is truly effective or not, as a single study may incorrectly show a drug to be effective simply by chance.

However, Hattie goes one step further and carries out a meta-analysis on other meta-analyses, forming a sort of "meta-meta-analysis". With this approach, his team only directly work with 400 articles, as each of these is a meta-analysis of tens or hundreds of other articles, which is how we reach the gargantuan number 80 thousand.

You would have thought that such an ambitious, influential, and widely praised work would have come under much careful scrutiny. And you would have thought that since it is so statistical, numerous other researchers in the field of education would have performed at least a surface-level plausibility check.

However, you may be disappointed. It took two years for anybody to even begin to notice the glaring statistical errors behind this work, and even when they were noticed, Hattie's team didn't treat them with the gravity they deserved. Methodological criticism gradually increased in number, and by now it is clear that the "Holy Grail" has numerous leaky holes.

In this episode, after introducing Visible Learning, I go on to take some highlights from one such criticism, entitled How to engage in pseudoscience with real data: A criticism of John Hattie’s arguments in Visible Learning from the perspective of a statistician, written by Canadian statistician Pierre-Jerome Bergeron. I aim to explain the most accessible points, and leave the more complex parts of the article for those with the interest and mathematical acumen to look up.

This sort of thing seems to happen a lot in education. It's one of the reasons why it's so hard to figure out how things work and what's actually true in the field. At least I can warn people about the problems with this still widely cited work.

Enjoy the episode.

#15 - Gordon Hibbard - Comfort and Growth Can't Coexist