Can Machines Understand? Benchmarking AI

Support the show to get full episodes, full archive, and join the Discord community.

The Transmitter is an online publication that aims to deliver useful information, insights and tools to build bridges across neuroscience and advance research. Visit thetransmitter.org to explore the latest neuroscience news and perspectives, written by journalists and scientists.

Read more about our partnership.

Sign up for Brain Inspired email alerts to be notified every time a new Brain Inspired episode is released.

To explore more neuroscience news and perspectives, visit thetransmitter.org.

Henk de Regt is a professor of Philosophy of Science and the director of the Institute for Science in Society at Radboud University. Henk wrote the book on Understanding. Literally, he wrote what has become a classic in philosophy of science, Understanding Scientific Understanding.

Henks' account of understanding goes roughly like this, but you can learn more in his book and other writings. To claim you understand something in science requires that you can produce a theory-based explanation of whatever you claim to understand, and it depends on you having the right scientific skills to be able to work productively with that theory - for example, making qualitative predictions about it without performing calculations. So understanding is contextual and depends on the skills of the understander.

There's more nuance to it, so like I said you should read the book, but this account of understanding distinguishes it from explanation itself, and distinguishes it from other accounts of understanding, which take understanding to be either a personal subjective sense - that feeling of something clicking in your mind - or simply the addition of more facts about something.

In this conversation, we revisit Henk's work on understanding, and how it touches on many other topics, like realism, the use of metaphors, how public understanding differs from expert understanding, idealization and abstraction in science, and so on.

And, because Henk's kind of understanding doesn't depend on subjective awareness or things being true, he and his cohorts have begun working on whether there could be a benchmark for degrees of understanding, to possibly asses whether AI demonstrates understanding, and to use as a common benchmark for humans and machines.

• Google Scholar page
• Social: @henkderegt.bsky.social;  
• Book:
  • Understanding Scientific Understanding.
• Related papers
  • Towards a benchmark for scientific understanding in humans and machines
  • Metaphors as tools for understanding in science communication among experts and to the public
  • Two scientific perspectives on nerve signal propagation: how incompatible approaches jointly promote progress in explanatory understanding

0:00 - Intro 10:13 - Philosophy of explanation vs understanding 14:32 - Different accounts of understanding 20:29 - Henk's account of understanding 26:47 - What counts as intelligible? 34:09 - Hodgkin and Huxley alternative 37:54 - Familiarity vs understanding 44:42 - Measuring understanding 1:02:53 - Machine understanding 1:16:39 - Non-factive understanding 1:23:34 - Abstraction vs understanding 1:31:07 - Public understanding of science 1:41:35 - Reflections on the book