How do benchmarks and AutoRouter guide model choice?

Register for Founder University Japan’s Kickoff! https://luma.com/cm0x90mk

Today’s show:

Find out why AI is perfectly suited to legal tasks… despite being too fast for “billable hours”

On today’s TWiST, Alex takes a deep dive into LLM Law with Harvey AI co-founder/president Gabe Pereyra. It turns out, much like software development, doing legal work relies on learning specialized language and sifting through a dense and expansive corpus of information… making it an IDEAL use case for LLMs.

Buuuuut AI works so fast… the current structure of legal payments (billable hours) no longer applies. Find out how Harvey is working around these challenges — and what they plan to do if OpenAI decides to get into the legal game — in this fascinating interview.

THEN, Alex chats with another AI visionary — OpenRouter co-founder and CEO Alex Atallah — who allows developers to plug and play all the major models into their applications. Find out why specialized LLMs trained for “resourcefulness” are coming into fashion… why benchmarks and evals have become so crucial to the industry… AND whether we’ll end up spending as much on AI inference as the human workers it’s replacing… in this essential discussion.

PLUS Jason stops by for some more Founder Q’s!

Timestamps:

(0:00) Welcome back to TWiST!

(1:32) Alex kicks off the show with Harvey AI co-founder and President Gabe Pereyra to talk about applying AI to the legal profession

(4:52) How major international law firms (and their enterprise clients) fueled Harvey’s mega-growth

(6:54) Why LLMs are perfectly suited for legal work

(9:24) Enterpret - Enterpret turns feedback noise into Customer Intelligence, so your team knows exactly what to fix and build next. Head to Enterpret.com/twist to book a demo and see it in action.x and build next. Head to Enterpret.com/twist to book a demo and see it in action.

(17:22) Looking ahead: expanding beyond the law.

(20:42) Uber AI Solutions - Your trusted partner to get AI to work in the real world. Book a demo with them TODAY at Uber.com/twist

(21:44) The problem: AI can’t charge “billable hours”… it’s too fast!

(23:44) Will the major AI players like Google and OpenAI build their own version of this?

(29:53) Lemon.io - Get 15% off your first 4 weeks of developer time at https://Lemon.io/twist

(30:41) Alex welcomes OpenRouter co-founder and CEO Alex Atallah, whose rankings he loves and uses all the time

(33:10) Why Alex says we’re seeing more specialized LLMs trained for more “resourcefulness”

(36:05) How increased competition and diversification makes things tougher for developers

(41:09) Alex suggests that new models are exciting for users, like product launches used to be

(42:29) OK, let’s get into it… How does OpenRouter make money?

(45:38) Understanding what kinds of training data the AI companies want the MOST

(47:47) The important currency of evals and benchmarks

(49:13) Why OpenRouter is ramping up its LLM recommendation engine

(53:44) Will AI inference spend end up costing as much as the humans it’s replacing?

(56:49) Time for a Founder Q! What should founders do about quick vibe-coded competitors?

Subscribe to the TWiST500 newsletter: https://ticker.thisweekinstartups.com

Check out the TWIST500: https://www.twist500.com

Subscribe to This Week in Startups on Apple: https://rb.gy/v19fcp

Follow Lon:

X: https://x.com/lons

Follow Alex:

X: https://x.com/alex

LinkedIn: ⁠https://www.linkedin.com/in/alexwilhelm

Thank you to our partners:

(9:24) Enterpret - Enterpret turns feedback noise into Customer Intelligence, so your team knows exactly what to fix and build next. Head to Enterpret.com/twist to book a demo and see it in action.x and build next. Head to Enterpret.com/twist to book a demo and see it in action.

(20:42) Uber AI Solutions - Your trusted partner to get AI to work in the real world. Book a demo with them TODAY at Uber.com/twist

(29:53) Lemon.io - Get 15% off your first 4 weeks of developer time at https://Lemon.io/twist