"The Cognitive Revolution" | AI Builders, Researchers, and Live Player Analysis

Everything You Wanted to Know About LLM Post-Training, with Nathan Lambert of Allen Institute for AI

Nov 21, 2024

Nathan Lambert, a machine learning researcher at the Allen Institute for AI and author of the Interconnex newsletter, dives into cutting-edge post-training techniques for large language models. He discusses the Tulu project, which enhances model performance through innovative methods like supervised fine-tuning and reinforcement learning. Lambert sheds light on the significance of human feedback, the challenges of data contamination, and the collaborative nature of AI research. His insights will resonate with anyone interested in the future of AI and model optimization.

01:50:08

Creator website

Episode guests

Nathan Lambert

AI Summary

AI Chapters

Episode notes

Podcast summary created with Snipd AI

Quick takeaways

Nathan Lambert emphasizes the importance of high-quality datasets in achieving superior performance in large language models (LLMs) over excessive preference tuning.

The challenges of ensuring character consistency across different models highlight the complexities of training AI in a politically influenced data landscape.

Data brokers' collection of personal information raises significant privacy concerns affecting both individuals and industries, underscoring the need for data reclamation services.

Innovative post-training methodologies, including reinforcement learning with verifiable rewards, demonstrate potential advancements in LLM capabilities and reasoning skills.

Deep dives

The Focus on Data Over Preference Tuning

Emphasizing the importance of improving data quality and pipelines over excessive preference tuning, it is argued that better training outcomes can be achieved with enhanced datasets. The discussion highlights the significant advancements observed with Meta's llama report and the rise of openAI and Google's scores, suggesting a competitive environment where data plays a critical role. The idea is to prioritize human involvement for comprehensive problem-solving while automating simpler tasks through large language models (LLMs). By shifting focus to creating high-quality data, opportunities for breakthroughs in model performance are vast.

Intro

6min

Innovations at the Allen Institute for AI

12min

Navigating E-Commerce Solutions and Cloud Innovations

3min

Optimizing Language Model Training

14min

Exploring Career Impact and AI Automation Tools

3min

Enhancing Language Model Evaluation and Performance Metrics

4min

Contamination and Tuning in Language Models

19min

Reinforcement Learning and Language Models

19min

Exploring Post-Training Dimensions and Model Character Development

2min

10.

Navigating Synthetic Data and LLM Evaluation

22min

11.

Exploring Robotics and AI Models

7min

In this episode of The Cognitive Revolution, we dive deep into frontier post-training techniques for large language models with Nathan Lambert from the Allen Institute for AI. Nathan discusses the groundbreaking Tulu 3 release, which matches Meta's post-training performance using the LlAMA base model. We explore supervised fine-tuning, preference-based reinforcement learning, and the innovative reinforcement learning from verifiable reward technique. Nathan provides unprecedented insights into the practical aspects of model development, compute requirements, and data generation strategies. This technically rich conversation illuminates previously opaque aspects of LLM development, achieved by a small team of 10-15 people. Join us for one of our most detailed and valuable discussions on state-of-the-art AI model development.

Check out Nathan's Lambert newsletter:

https://www.natolambert.com

https://www.interconnects.ai

Be notified early when Turpentine's drops new publication: https://www.turpentine.co/exclusiveaccess

SPONSORS:

Incogni: Take your personal data back with Incogni! Use code REVOLUTION at the link below and get 60% off an annual plan: https://incogni.com/revolution

Notion: Notion offers powerful workflow and automation templates, perfect for streamlining processes and laying the groundwork for AI-driven automation. With Notion AI, you can search across thousands of documents from various platforms, generating highly relevant analysis and content tailored just for you - try it for free at https://notion.com/cognitiverevolution

Shopify: Shopify is the world's leading e-commerce platform, offering a market-leading checkout system and exclusive AI apps like Quikly. Nobody does selling better than Shopify. Get a $1 per month trial at https://shopify.com/cognitive

Oracle Cloud Infrastructure (OCI): Oracle's next-generation cloud platform delivers blazing-fast AI and ML performance with 50% less for compute and 80% less for outbound networking compared to other cloud providers13. OCI powers industry leaders with secure infrastructure and application development capabilities. New U.S. customers can get their cloud bill cut in half by switching to OCI before December 31, 2024 at https://oracle.com/cognitive

80,000 Hours: 80,000 Hours offers free one-on-one career advising for Cognitive Revolution listeners aiming to tackle global challenges, especially in AI. They connect high-potential individuals with experts, opportunities, and personalized career plans to maximize positive impact. Apply for a free call at https://80000hours.org/cognitiverevolution to accelerate your career and contribute to solving pressing AI-related issues.

RECOMMENDED PODCAST:

Unpack Pricing - Dive into the dark arts of SaaS pricing with Metronome CEO Scott Woody and tech leaders. Learn how strategic pricing drives explosive revenue growth in today's biggest companies like Snowflake, Cockroach Labs, Dropbox and more.

Apple: https://podcasts.apple.com/us/podcast/id1765716600

Spotify: https://open.spotify.com/show/38DK3W1Fq1xxQalhDSueFg

CHAPTERS:

(00:00:00) Teaser

(00:00:59) Sponsors: Incogni

(00:02:20) About the Episode

(00:05:56) Introducing AI2

(00:09:56) Tulu: Deep Dive (Part 1)

(00:17:43) Sponsors: Shopify | Oracle Cloud Infrastructure (OCI)

(00:20:38) Open vs. Closed Recipes

(00:29:48) Compute & Value (Part 1)

(00:34:22) Sponsors: 80,000 Hours | Notion

(00:37:02) Compute & Value (Part 2)

(00:42:41) Model Weight Evolution

(00:53:16) DPO vs. PPO

(01:06:36) Project Trajectory

(01:20:39) Synthetic Data & LLM Judge

(01:27:39) Verifiable RL

(01:38:17) Advice for Practitioners

(01:44:01) Open Source vs. Closed

(01:49:18) Outro

Get the Snipd
podcast app

Unlock the knowledge in podcasts with the podcast player of the future.

AI-powered
podcast player

Listen to all your favourite podcasts with AI-powered features

Discover
highlights

Listen to the best highlights from the podcasts you love and dive into the full episode

Save any
moment

Hear something you like? Tap your headphones to save it with AI-generated key takeaways

Share
& Export

Send highlights to Twitter, WhatsApp or export them to Notion, Readwise & more