Evaluating Reasoning in Language Models

Our 213nd episode with a summary and discussion of last week's big AI news!
Recorded on 06/21/2025

Hosted by Andrey Kurenkov and Jeremie Harris.
Feel free to email us your questions and feedback at contact@lastweekinai.com and/or hello@gladstone.ai

Read out our text newsletter and comment on the podcast at https://lastweekin.ai/.

In this episode:

• Midjourney launches its first AI video generation model, moving from text-to-image to video with a subscription model offering up to 21-second clips, highlighting the affordability and growing capabilities in AI video generation.
• Google's Gemini AI family updates include high-efficiency models for cost-effective workloads, and new enhancements in Google's search function now allow for voice interactions.
• The introduction of two new benchmarks, Live Code Bench Pro and Abstention Bench, aiming to test and improve the problem-solving and abstention capabilities of reasoning models, revealing current limitations.
• OpenAI wins a $200 million US defense contract to support various aspects of the Department of Defense, reflecting growing collaborations between tech companies and government for AI applications.

Timestamps + Links:

• (00:00:10) Intro / Banter
• (00:01:32) News Preview
  
• Tools & Apps

• (00:02:12) Midjourney launches its first AI video generation model, V1
• (00:05:52) Google’s Gemini AI family updated with stable 2.5 Pro, super-efficient 2.5 Flash-Lite
• (00:07:59) Google’s AI Mode can now have back-and-forth voice conversations
• (00:10:13) YouTube to Add Google’s Veo 3 to Shorts in Move That Could Turbocharge AI on the Video Platform

• Applications & Business

• (00:11:10) The ‘OpenAI Files’ will help you understand how Sam Altman’s company works
• (00:12:29) OpenAI drops Scale AI as a data provider following Meta deal
• (00:13:28) Amazon’s Zoox opens its first major robotaxi production facility

• Projects & Open Source

• (00:15:20) LiveCodeBench Pro: How Do Olympiad Medalists Judge LLMs in Competitive Programming?
• (00:19:45) AbstentionBench: Reasoning LLMs Fail on Unanswerable Questions
• (00:22:49) MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention

• Research & Advancements

• (00:24:33) Scaling Laws of Motion Forecasting and Planning -- A Technical Report

• Policy & Safety

• (00:28:07) Universal Jailbreak Suffixes Are Strong Attention Hijackers
• (00:30:52) OpenAI found features in AI models that correspond to different ‘personas’
• (00:33:25) OpenAI wins $200 million U.S. defense contract