The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence)

Is ChatGPT Getting Worse? with James Zou - #645

Sep 4, 2023

In this conversation, James Zou, an assistant professor at Stanford known for his work in biomedical data science, dives into the evolving landscape of ChatGPT. He examines its fluctuating performance over recent months, discussing intriguing comparisons between versions. The potential for surgical AI enhancements inspires thoughts on the future of large language models. Zou also shares innovative insights on using Twitter data to build medical imaging datasets, addressing the challenges of data quality and oversight in AI for healthcare applications.

42:17

Creator website

Episode guests

James Zou

AI Summary

Highlights

AI Chapters

Episode notes

Podcast summary created with Snipd AI

Quick takeaways

The performance of ChatGPT varied significantly across different tasks, highlighting the need for more precise edits and control in large language models.

Leveraging data from social media platforms like Twitter can be valuable for training AI systems in specific domains, such as pathology image analysis.

Deep dives

Chet GPT and Changes in Behavior

The podcast episode discusses the research on the change in behavior of Chet GPT, a popular language model. The researchers conducted a systematic assessment by comparing the performance of Chet GPT in March and June versions across various tasks. Surprisingly, they found that while the later version performed better in some tasks, it performed substantially worse in others, including seemingly simpler tasks like identifying prime numbers. The researchers explored possible reasons for this change in behavior, such as conflicting objectives and the concept of "pleiotropy". They highlighted the need for more precise and surgical edits to large language models to improve control and transparency.

Challenges in Evaluating Text Output of Language Models

01:40

The Mystery of Proprietary AI Models: Performance Decreases and Neuro Playotropy

01:14

Using Social Media to Curate Large Data Sets for Medical Language Models

01:55

Intro

2min

Evaluating ChatGPT's Evolution

10min

Unraveling AI Complexity: The Paradoxes of Safety Enhancements

2min

Balancing Safety and Instruction in AI Models

6min

The Precision of Language Models

4min

Monitoring Language Model Behavior

3min

Innovative Data Curation in Medical Imaging

15min

Today we’re joined by James Zou, an assistant professor at Stanford University. In our conversation with James, we explore the differences in ChatGPT’s behavior over the last few months. We discuss the issues that can arise from inconsistencies in generative AI models, how he tested ChatGPT’s performance in various tasks, drawing comparisons between March 2023 and June 2023 for both GPT-3.5 and GPT-4 versions, and the possible reasons behind the declining performance of these models. James also shared his thoughts on how surgical AI editing akin to CRISPR could potentially revolutionize LLM and AI systems, and how adding monitoring tools can help in tracking behavioral changes in these models. Finally, we discuss James' recent paper on pathology image analysis using Twitter data, in which he explores the challenges of obtaining large medical datasets and data collection, as well as detailing the model’s architecture, training, and the evaluation process.

The complete show notes for this episode can be found at twimlai.com/go/645.

Remember Everything You Learn from Podcasts

Save insights instantly, chat with episodes, and build lasting knowledge - all powered by AI.

The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence)

Is ChatGPT Getting Worse? with James Zou - #645

Episode guests

Podcast summary created with Snipd AI

Quick takeaways

Deep dives

Chet GPT and Changes in Behavior

Visual Language Model for Pathology

Leveraging Social Media for Data Collection

Implications and Challenges

Challenges in Evaluating Text Output of Language Models

The Mystery of Proprietary AI Models: Performance Decreases and Neuro Playotropy

Using Social Media to Curate Large Data Sets for Medical Language Models

Remember Everything You Learn from Podcasts