Benchmarks, Gaming, and 'Benchmaxxing'

Today’s show:

The race for more compute gets all the headlines, but Mistral AI CEO Arthur Mensch tells us the REAL danger is running out of data!

On TWiST, Jason and Alex are chatting with the head of France’s largest foundation AI company, Mistral AI, about how enterprises can get more out of their AI pilots, whether the West is lacking leadership on open-source, why Mistral AI isn’t pursuing every possible kind of AI application, and optimizing your models for performance rather than benchmarks (aka benchmaxxing). Find out what it’s like to run a huge AI company in today’s intense environment!

PLUS… why everyone made the wrong call on OpenAI vs. Google… Kraken’s fresh $20 billion valuation… Jason’s urgent message to every company thinking about M&A… and much more!

Timestamps:

(3:35) Why did everyone get it wrong on Google vs. OpenAI?

(10:51) Why Jason thinks drone delivery will make everyone safer

(9:52) SQUARESPACE: Use offer code TWIST to save 10% off your first purchase of a website or domain at https://www.Squarespace.com/TWIST

(11:26) Mistral AI CEO Arthur Mensch joins us! What’s it like to run a foundation model AI company in this intense environment?

(14:50) Why enterprises aren’t getting enough value out of AI and how Mistral AI aims to correct this

(18:45) Why Mistral AI’s goal is, ultimately, to disappear…

(20:19) LinkedIn Ads: Start converting your B2B audience into high quality leads today. Launch your first campaign and get $250 FREE when you spend at least $250. Go to http://linkedin.com/thisweekinstartups to claim your credit.

(22:49) Are Europe and the US lacking leadership on open source AI?

(24:53) Mistral AI doesn’t need to compete in every category... how big of an advantage is focus?

(27:05) The “two bottlenecks” in Machine Learning: Compute and Data

(29:01) How Mistral AI works with human experts to annotate their models, and where they source them from

(30:06) Nexos.ai: Stop Shadow AI in its tracks with the unified platform for secure AI adoption and productivity. Try it with a free 14-day trial at https://nexos.ai/twist.

(34:24) Is optimizing your model for benchmark tests (aka benchmaxxing) like cheating?

(40:08) Will we ever be able to run smaller AI models on our phones?

(44:21) When does Arthur think we’ll have truly helpful humanoid robots in our lives?

(49:42) Suno lawsuits and why Jason says you should NEVER break the rules, ESPECIALLY with the music industry

(54:28) Release the Kraken! They’re now worth $20 billion.

(56:48) Jason is going to F1… should he take up the offer and do a dry run?

(58:30) How Meta beat the FTC and why Jason is so excited about it

(1:01:37) Who should buy Crunchbase?