Building Trust and Rigor in Biomedical AI

Episode 126

I spoke with Vivek Natarajan about:

* Improving access to medical knowledge with AI

* How an LLM for medicine should behave

* Aspects of training Med-PaLM and AMIE

* How to facilitate appropriate amounts of trust in users of medical AI systems

Vivek Natarajan is a Research Scientist at Google Health AI advancing biomedical AI to help scale world class healthcare to everyone. Vivek is particularly interested in building large language models and multimodal foundation models for biomedical applications and leads the Google Brain moonshot behind Med-PaLM, Google's flagship medical large language model. Med-PaLM has been featured in The Scientific American, The Economist, STAT News, CNBC, Forbes, New Scientist among others.

I spend a lot of time on this podcast—if you like my work, you can support me on Patreon :)

Reach me at editor@thegradient.pub for feedback, ideas, guest suggestions.

Subscribe to The Gradient Podcast:  Apple Podcasts  | Spotify | Pocket Casts | RSSFollow The Gradient on Twitter

Outline:

* (00:00) Intro

* (00:35) The concept of an “AI doctor”

* (06:54) Accessibility to medical expertise

* (10:31) Enabling doctors to do better/different work

* (14:35) Med-PaLM

* (15:30) Instruction tuning, desirable traits in LLMs for medicine

* (23:41) Axes for evaluation of medical QA systems

* (30:03) Medical LLMs and scientific consensus

* (35:32) Demographic data and patient interventions

* (40:14) Data contamination in Med-PaLM

* (42:45) Grounded claims about capabilities

* (45:48) Building trust

* (50:54) Genetic Discovery enabled by a LLM

* (51:33) Novel hypotheses in genetic discovery

* (57:10) Levels of abstraction for hypotheses

* (1:01:10) Directions for continued progress

* (1:03:05) Conversational Diagnostic AI

* (1:03:30) Objective Structures Clinical Examination as an evaluative framework

* (1:09:08) Relative importance of different types of data

* (1:13:52) Self-play — conversational dispositions and handling patients

* (1:16:41) Chain of reasoning and information retention

* (1:20:00) Performance in different areas of medical expertise

* (1:22:35) Towards accurate differential diagnosis

* (1:31:40) Feedback mechanisms and expertise, disagreement among clinicians

* (1:35:26) Studying trust, user interfaces

* (1:38:08) Self-trust in using medical AI models

* (1:41:39) UI for medical AI systems

* (1:43:50) Model reasoning in complex scenarios

* (1:46:33) Prompting

* (1:48:41) Future outlooks

* (1:54:53) Outro

Links:

* Vivek’s Twitter and homepage

* Papers

* Towards Expert-Level Medical Question Answering with LLMs (2023)

* LLMs encode clinical knowledge (2023)

* Towards Generalist Biomedical AI (2024)

* AMIE

* Genetic Discovery enabled by a LLM (2023)