The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence)

Is ChatGPT Getting Worse? with James Zou - #645

Sep 4, 2023

42:17

Snipd AI

In this episode, James Zou, an assistant professor at Stanford University, discusses the changing behavior of ChatGPT, comparing GPT-3.5 and GPT-4 versions. He also shares insights on CRISPR's impact on LLM and AI systems, monitoring behavioral changes in models, and using Twitter data for pathology image analysis.

AI Summary

Highlights

AI Chapters

Episode notes

Podcast summary created with Snipd AI

Quick takeaways

The performance of ChatGPT varied significantly across different tasks, highlighting the need for more precise edits and control in large language models.

Leveraging data from social media platforms like Twitter can be valuable for training AI systems in specific domains, such as pathology image analysis.

Deep dives

Chet GPT and Changes in Behavior

The podcast episode discusses the research on the change in behavior of Chet GPT, a popular language model. The researchers conducted a systematic assessment by comparing the performance of Chet GPT in March and June versions across various tasks. Surprisingly, they found that while the later version performed better in some tasks, it performed substantially worse in others, including seemingly simpler tasks like identifying prime numbers. The researchers explored possible reasons for this change in behavior, such as conflicting objectives and the concept of "pleiotropy". They highlighted the need for more precise and surgical edits to large language models to improve control and transparency.

Challenges in Evaluating Text Output of Language Models

01:40

The Mystery of Proprietary AI Models: Performance Decreases and Neuro Playotropy

01:14

Using Social Media to Curate Large Data Sets for Medical Language Models

01:55

Introduction

2min

Analyzing the Changing Behavior of Chat GPT

2min

Evaluating the Output of Text-based Language Models

8min

Decoupling Safety and Control in Large Language Models

12min

Implications for Developers and Engineers

3min

Visual Language Models for Medical Twitter

7min

Training a Model for Aligning Vision Encoders with Language Encoders in Medical Contexts

8min

Today we’re joined by James Zou, an assistant professor at Stanford University. In our conversation with James, we explore the differences in ChatGPT’s behavior over the last few months. We discuss the issues that can arise from inconsistencies in generative AI models, how he tested ChatGPT’s performance in various tasks, drawing comparisons between March 2023 and June 2023 for both GPT-3.5 and GPT-4 versions, and the possible reasons behind the declining performance of these models. James also shared his thoughts on how surgical AI editing akin to CRISPR could potentially revolutionize LLM and AI systems, and how adding monitoring tools can help in tracking behavioral changes in these models. Finally, we discuss James' recent paper on pathology image analysis using Twitter data, in which he explores the challenges of obtaining large medical datasets and data collection, as well as detailing the model’s architecture, training, and the evaluation process.

The complete show notes for this episode can be found at twimlai.com/go/645.

Get the Snipd
podcast app

Unlock the knowledge in podcasts with the podcast player of the future.

AI-powered
podcast player

Listen to all your favourite podcasts with AI-powered features

Discover
highlights

Listen to the best highlights from the podcasts you love and dive into the full episode

Save any
moment

Hear something you like? Tap your headphones to save it with AI-generated key takeaways

Share
& Export