Gamifying AI Model Evaluation

Hey all, Alex here 👋

This week, while not the busiest week in releases (we can't get a SOTA LLM every week now can we), was full of interesting open source releases, and feature updates such as the chatGPT meetings recorder (which we live tested on the show, the limit is 2 hours!)

It was also a day after our annual W&B conference called FullyConnected, and so I had a few goodies to share with you, like answering the main question, when will W&B have some use of those GPUs from CoreWeave, the answer is... now! (We launched a brand new preview of an inference service with open source models)

And finally, we had a great chat with Pankaj Gupta, co-founder and CEO of Yupp, a new service that lets users chat with the top AIs for free, while turning their votes into leaderboards for everyone else to understand which Gen AI model is best for which task/topic. It was a great conversation, and he even shared an invite code with all of us (I'll attach to the TL;DR and show notes, let's dive in!)

00:00 Introduction and Welcome

01:04 Show Overview and Audience Interaction

01:49 Special Guest Announcement and Experiment

03:05 Wolfram's Background and Upcoming Hosting

04:42 TLDR: This Week's Highlights

15:38 Open Source AI Releases

32:34 Big Companies and APIs

32:45 Google's Gemini Updates

42:25 OpenAI's Latest Features

54:30 Exciting Updates from Weights & Biases

56:42 Introduction to Weights & Biases Inference Service

57:41 Exploring the New Inference Playground

58:44 User Questions and Model Recommendations

59:44 Deep Dive into Model Evaluations

01:05:55 Announcing Online Evaluations via Weave

01:09:05 Introducing Pankaj Gupta from YUP.AI

01:10:23 YUP.AI: A New Platform for Model Evaluations

01:13:05 Discussion on Crowdsourced Evaluations

01:27:11 New Developments in Video Models

01:36:23 OpenAI's New Transcription Service

01:39:48 Show Wrap-Up and Future Plans

Here's the TL;DR and show notes links

ThursdAI - June 19th, 2025 - TL;DR

* Hosts and Guests

* Alex Volkov - AI Evangelist & Weights & Biases (@altryne)

* Co Hosts - @WolframRvnwlf @yampeleg @nisten @ldjconfirmed

* Guest - @pankaj - co-founder of Yupp.ai

* Open Source LLMs

* Moonshot AI open-sourced Kimi-Dev-72B (Github, HF)

* MiniMax-M1 456B (45B Active) - reasoning model (Paper, HF, Try It, Github)

* Big CO LLMs + APIs

* Google drops Gemini 2.5 Pro/Flash GA, 2.5 Flash-Lite in Preview ( Blog, Tech report, Tweet)

* Google launches Search Live: Talk, listen and explore in real time with AI Mode (Blog)

* OpenAI adds MCP support to Deep Research in chatGPT (X, Docs)

* OpenAI launches their meetings recorder in mac App (docs)

* Zuck update: Considering bringing Nat Friedman and Daniel Gross to Meta (information)

* This weeks Buzz

* NEW! W&B Inference provides a unified interface to access and run top open-source AI models (inference, docs)

* NEW! W&B Weave Online Evaluations delivers real-time production insights and continuous evaluation for AI agents across any cloud. (X)

* The new platform offers "metal-to-token" observability, linking hardware performance directly to application-level metrics.

* Vision & Video

* ByteDance new video model beats VEO3 - Seedance.1.0 mini (Site, FAL)

* MiniMax Hailuo 02 - 1080p native, SOTA instruction following (X, FAL)

* Midjourney video is also here - great visuals (X)

* Voice & Audio

* Kyutai launches open-source, high-throughput streaming Speech-To-Text models for real-time applications (X, website)

* Studies and Others

* LLMs Flunk Real-World Coding Contests, Exposing a Major Skill Gap (Arxiv)

* MIT Study: ChatGPT Use Causes Sharp Cognitive Decline (Arxiv)

* Andrej Karpathy's "Software 3.0": The Dawn of English as a Programming Language (youtube, deck)

* Tools

* Yupp launches with 500+ AI models, a new leaderboard, and a user-powered feedback economy - use thursdai link* to get 50% extra credits

* BrowserBase announces director.ai - an agent to run things on the web

* Universal system prompt for reduction of hallucination (from Reddit)

*Disclosure: while this isn't a paid promotion, I do think that yupp has a great value, I do get a bit more credits on their platform if you click my link and so do you. You can go to yupp.ai and register with no affiliation if you wish.

This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit sub.thursdai.news/subscribe