Benchmarks vs. real scientific work

Jungwon argues benchmarks miss messy, tacit, judgment-heavy research tasks that real scientists face daily.

Play episode from 01:44:24

chevron_right

Transcript

chevron_right

Transcript

Episode notes

This year-end live show features nine rapid-fire conversations to make sense of AI’s 2025 and what might define 2026. PSA for AI builders: Interested in alignment, governance, or AI safety? Learn more about the MATS Summer 2026 Fellowship and submit your name to be notified when applications open: https://matsprogram.org/s26-tcr. Zvi Moshowitz maps the OpenAI–Anthropic–Google race, the denialism gap, and why his PDoom is still ~60–70%. Greg (ARC-AGI Prize), Eugenia Kuyda, Ali Behrouz, Logan Kirkpatrick, and Jungwon Hwang cover sample-efficient benchmarks and ARC-AGI 3, companions and human-flourishing metrics, continual-learning memory, Gemini 3 Flash for developers, and AI for scientific decisions.

Sponsors:

Gemini 3 in Google AI Studio:

Gemini 3 in Google AI Studio lets you build fully functional apps from a simple description—no coding required. Start vibe coding your idea today at https://ai.studio/build

MATS:

MATS is a fully funded 12-week research program pairing rising talent with top mentors in AI alignment, interpretability, security, and governance. Apply for the next cohort at https://matsprogram.org/s26-tcr

Framer:

Framer is the all-in-one tool to design, iterate, and publish stunning websites with powerful AI features. Start creating for free and use code COGNITIVE to get one free month of Framer Pro at https://framer.com/design

Shopify:

Shopify powers millions of businesses worldwide, handling 10% of U.S. e-commerce. With hundreds of templates, AI tools for product descriptions, and seamless marketing campaign creation, it's like having a design studio and marketing team in one. Start your $1/month trial today at https://shopify.com/cognitive

Tasklet:

Tasklet is an AI agent that automates your work 24/7; just describe what you want in plain English and it gets the job done. Try it for free and use code COGREV for 50% off your first month at https://tasklet.ai

CHAPTERS:

(00:00) Sponsor: Gemini 3 in Google AI Studio

(00:31) Live show experiment

(02:26) Zvi: discourse and denial

(13:28) Continual learning and doom

(22:05) ArcAGI: what's missing (Part 1)

(22:09) Sponsors: MATS | Framer

(25:28) ArcAGI: what's missing (Part 2)

(31:58) Scaffolds and tiny models

(38:58) ArcAGI 3 game worlds

(45:13) AI companions landscape (Part 1)

(45:21) Sponsors: Shopify | Tasklet