Evaluating AI: Insights from Gaming and Coding

This chapter explores the complexities of comparing AI models and emphasizes the necessity of effective prompting for evaluation. It draws parallels between the strategic mindset in card games like Magic: The Gathering and coding, showcasing the differences in human intuition and AI capabilities.

Transcript

chevron_right

Play full episode

chevron_right

Transcript

Episode notes

In this episode of The Cognitive Revolution, Nathan welcomes back Zvi Mowshowitz for an in-depth discussion on the latest developments in AI over the past six months. They explore Ilya's new superintelligence-focused startup, analyze OpenAI's O1 model, and debate the impact of Claude's computer use capabilities. The conversation covers emerging partnerships in big tech, regulatory changes, and the recent OpenAI profit-sharing drama. Zvi offers unique insights on AI safety, politics, and strategic analysis that you won't find elsewhere. Join us for this thought-provoking episode that challenges our understanding of the rapidly evolving AI landscape.

Check out "Don't Worry About the Vase" Blog: https://thezvi.substack.com

Be notified early when Turpentine's drops new publication: https://www.turpentine.co/exclusiveaccess

SPONSORS:

Shopify: Shopify is the world's leading e-commerce platform, offering a market-leading checkout system and exclusive AI apps like Quikly. Nobody does selling better than Shopify. Get a $1 per month trial at https://shopify.com/cognitive

Notion: Notion offers powerful workflow and automation templates, perfect for streamlining processes and laying the groundwork for AI-driven automation. With Notion AI, you can search across thousands of documents from various platforms, generating highly relevant analysis and content tailored just for you - try it for free at https://notion.com/cognitiverevolution

Oracle Cloud Infrastructure (OCI): Oracle's next-generation cloud platform delivers blazing-fast AI and ML performance with 50% less for compute and 80% less for outbound networking compared to other cloud providers13. OCI powers industry leaders with secure infrastructure and application development capabilities. New U.S. customers can get their cloud bill cut in half by switching to OCI before December 31, 2024 at https://oracle.com/cognitive

SelectQuote: Finding the right life insurance shouldn't be another task you put off. SelectQuote compares top-rated policies to get you the best coverage at the right price. Even in our AI-driven world, protecting your family's future remains essential. Get your personalized quote at https://selectquote.com/cognitive

RECOMMENDED PODCAST:

Unpack Pricing - Dive into the dark arts of SaaS pricing with Metronome CEO Scott Woody and tech leaders. Learn how strategic pricing drives explosive revenue growth in today's biggest companies like Snowflake, Cockroach Labs, Dropbox and more.

Apple: https://podcasts.apple.com/us/podcast/id1765716600

Spotify: https://open.spotify.com/show/38DK3W1Fq1xxQalhDSueFg

CHAPTERS:

(00:00:00) Teaser

(00:01:03) About the Episode

(00:02:57) Catching Up

(00:04:00) Ilya's New Company

(00:06:10) GPT-4 and Scaling

(00:11:49) User Report: GPT-4 (Part 1)

(00:18:11) Sponsors: Shopify | Notion

(00:21:06) User Report: GPT-4 (Part 2)

(00:24:25) Magic: The Gathering (Part 1)