The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence) Is ChatGPT Getting Worse? with James Zou - #645
11 snips
Sep 4, 2023 In this conversation, James Zou, an assistant professor at Stanford known for his work in biomedical data science, dives into the evolving landscape of ChatGPT. He examines its fluctuating performance over recent months, discussing intriguing comparisons between versions. The potential for surgical AI enhancements inspires thoughts on the future of large language models. Zou also shares innovative insights on using Twitter data to build medical imaging datasets, addressing the challenges of data quality and oversight in AI for healthcare applications.
AI Snips
Chapters
Transcript
Episode notes
ChatGPT's Evolving Behavior
- ChatGPT's behavior changes over time, impacting various applications.
- This necessitates systematic assessment across diverse tasks to understand these shifts.
Evaluating Complex Output
- Evaluating ChatGPT's complex output requires diverse methods.
- Prime number identification illustrates how accuracy can be assessed for simpler tasks.
Surprising Performance Drop
- Surprisingly, later ChatGPT versions performed worse on some tasks, including prime number identification.
- Chain-of-thought reasoning, effective in March, declined in June, impacting accuracy.