Benchmarking Gemini Ultra and Gemini Pro

17min Snip

00:00

Play full episode

Summary

Transcript

Episode notes

The speakers compare the performance of different language models, including Gemini and GPT-4, on various benchmarks such as GSM 8K and MM LU. They discuss the higher scores achieved by Gemini Ultra compared to GPT-4 and speculate on the reasons for the differences. They highlight the implications of these benchmark results on the structure of language models.

This Week in Startups is brought to you by…

Embroker. The Embroker Startup Insurance Program helps startups secure the most important types of insurance at a lower cost and with less hassle. Save up to 20% off of traditional insurance today at Embroker.com/twist. While you’re there, get an extra 10% off using offer code TWIST. Squarespace. Turn your idea into a new website! Go to Squarespace.com/TWIST for a free trial. When you’re ready to launch, use offer code TWIST to save 10% off your first purchase of a website or domain. Northwest Registered Agent. When starting your business, it's important to use a service that will actually help you. Northwest Registered Agent is that service. They'll form your company fast, give you the documents you need to open a business bank account, and even provide you with mail scanning and a business address to keep your personal privacy intact. Visit http://northwestregisteredagent.com/twist to get a 60% discount on your next LLC.

Today’s show:

Sunny Madra joins Jason to demo Google’s NotebookLM (5:00), Bard’s new capabilities after Gemini upgrade (17:13), Mixtral’s 8x7B model (53:59), and much more!

TIMESTAMPS

(0:00) Sunny Madra joins Jason (5:00) Sunny demos Google NotebookLM (11:22) Embroker - Use code TWIST to get an extra 10% off insurance at https://Embroker.com/twist (17:13) Bard's Gemini upgrade and Google's branding challenges (24:45) Ars Technica's experimental test bed (28:17) Squarespace - Use offer code TWIST to save 10% off your first purchase of a website or domain at https://Squarespace.com/twist (29:14) Unpacking Bard's enhanced capabilities (38:44) Northwest Registered Agent - Get a 60% discount on your next LLC at http://northwestregisteredagent.com/twist (39:50) Evaluating Gemini, Its market position, and the GPT Landscape (53:59) Sunny demos Mixtral's 8x7B model (1:07:38) Sunny creates an AI influencer using Juggernaut XL model in Google Colab (1:18:24) Sunny demos Magnific’s powerful image upscaler * LINKS: https://arstechnica.com/ai/2023/12/chatgpt-vs-google-bard-round-2-how-does-the-new-gemini-model-fare/https://twitter.com/sundarpichai/status/1732433036929589301 https://www.youtube.com/watch?v=K4pX1VAxaAI https://www.youtube.com/watch?v=kna9E_3kFF0&list=PL24nOpPUQlbYd1U349UDH2rrPaWWreM79&index=3 https://notebooklm.google.com/?pli=1 poe.com https://magnific.ai/upgrade/

Follow Sunny: https://twitter.com/sundeep

Follow Jason:

X: https://twitter.com/jason

Instagram: https://www.instagram.com/jason

LinkedIn: https://www.linkedin.com/in/jasoncalacanis

Great 2023 interviews: Steve Huffman, Brian Chesky, Aaron Levie, Sophia Amoruso, Reid Hoffman, Frank Slootman, Billy McFarland

Check out Jason’s suite of newsletters: https://substack.com/@calacanis

Follow TWiST:

Substack: https://twistartups.substack.com

Twitter: https://twitter.com/TWiStartups

YouTube: https://www.youtube.com/thisweekin

Subscribe to the Founder University Podcast: https://www.founder.university/podcast