Exploring the SWE-perf Benchmark for Code Optimization

Our 218th episode with a summary and discussion of last week's big AI news!
Recorded on 07/25/2025

Hosted by Andrey Kurenkov and Jeremie Harris.
Feel free to email us your questions and feedback at contact@lastweekinai.com and/or hello@gladstone.ai

Read out our text newsletter and comment on the podcast at https://lastweekin.ai/.

In this episode:

• GitHub introduces Vibe Coding with Spark, engaging users with natural language and visual controls to develop full-stack applications.
• AI coding tools from Gemin, CLI and RepleIt face significant issues, inadvertently deleting user data and highlighting the importance of careful management.
• US release never Award Americans, AI Action Plan outlining economic, technical, and policy strategies to maintain leadership in AI technology.
• Newly released Mega Science and SWE-Perf data sets evaluate AI reasoning and performance capabilities in diverse scientific and software engineering tasks.
  

Timestamps + Links:

• (00:00:10) Intro / Banter
• (00:01:31) News Preview
• Tools & Apps

• • (00:03:53) GitHub Introduces Vibe Coding with Spark: Revolutionizing Intelligent App Development in a Flash - MarkTechPost
  • (00:07:05) Figma’s AI app building tool is now available for everyone | The Verge
  • (00:10:18) Two major AI coding tools wiped out user data after making cascading mistakes - Ars Technica
  • (00:14:10) Google's AI Overviews have 2B monthly users, AI Mode 100M in the US and India | TechCrunch
• Applications & Business

• • (00:18:10) Leaked Memo: Anthropic CEO Says the Company Will Pursue Gulf State Investments After All
  • (00:24:39) Mira Murati says her startup Thinking Machines will release new product in ‘months’ with ‘significant open source component’
  • (00:27:07) Waymo responds to Tesla’s dick joke with a bigger Austin robotaxi map | The Verge
• Projects & Open Source

• • (00:32:05) MegaScience: Pushing the Frontiers of Post-Training Datasets for Science Reasoning
  • (00:43:09) TikTok Researchers Introduce SWE-Perf: The First Benchmark for Repository-Level Code Performance Optimization - MarkTechPost
• Research & Advancements

• • (00:47:17) Subliminal Learning: Language models transmit behavioral traits via hidden signals in data
  • (00:55:34) Inverse Scaling in Test-Time Compute
  • (01:02:34) Scaling Laws for Optimal Data Mixtures
• Policy & Safety

• • (01:07:35) White House Unveils America’s AI Action Plan
  • (01:16:55) Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety
  • (01:20:20) Self-preservation or Instruction Ambiguity? Examining the Causes of Shutdown Resistance
  • (01:24:00) People Are Being Involuntarily Committed, Jailed After Spiraling Into "ChatGPT Psychosis"
  • (01:28:03) Meta refuses to sign EU’s AI code of practice