Last Week in AI

#163 - Llama 3, Grok-1.5 Vision, new Atlas robot, RHO-1, Medium ban

Apr 24, 2024

Llama 3 and Grok-1.5 Vision claim the spotlight with their AI advancements. Boston Dynamics unveils a new Atlas robot for commercial use. US blacklists key partners, while Elon Musk hints at massive GPU requirements for Grok 3 model. Dr. Andrew Ng joins Amazon's board, and collaborative robotics secures a $100M investment. Exciting advancements in AI and robotics technology unfold in this episode.

01:33:54

Creator website

AI Summary

Highlights

AI Chapters

Episode notes

Podcast summary created with Snipd AI

Quick takeaways

Llama 3 by Meta touted as top open model, Grok-1.5 Vision overshadows GPT-4V, Reka Core competes with GPT-4

Boston Dynamics introduces new commercial Atlas robot, TSMC's $65B investment lacks complete chip puzzle, China blacklists Intel's key partner

Next-gen Grok 3 model requires 100,000 Nvidia H100 GPUs, Dr. Andrew Ng joins Amazon Board, $100M funding for Collaborative Robotics

Deep dives

Selective Language Modeling: Efficient Training on Useful Tokens

This paper introduces the concept of selective language modeling, focusing on training only on tokens that align with distribution instead of predicting every token. By selectively training on challenging tokens, the model optimizes performance more efficiently. They use a reference model to measure loss on each token, allowing for a more efficient training scheme, resulting in a 16% performance boost on benchmarks for 1 billion parameter models.

Optimizing Model Training with Compute Power and Data Size

01:18

Strategic Implications of Model Training Teasers

01:43

Introduction

2min

Llama 3 and AI Landscape Impact

28min

TSMC's $65 Billion US Investments and Export Control Challenges

8min

Elon Musk's Announcement on GRAQ-Free Model Training

2min

Advancements in Robotics and AI at Amazon and Beyond

11min

Exploring Selective Language Modeling for Efficient Model Optimization

4min

Efficient Training Strategies for Language Models

18min

AI Safety Institute Leadership Expansion and National Security Concerns

20min

Our 163rd episode with a summary and discussion of last week's big AI news!

Note: apology for this one coming out a few days late, got delayed in editing it -Andrey

Read out our text newsletter and comment on the podcast at https://lastweekin.ai/

Email us your questions and feedback at contact@lastweekin.ai and/or hello@gladstone.ai

Timestamps + links:

Intro / Banter
Tools & Apps
- (00:02:16) Meta releases Llama 3, claims it’s among the best open models available
- (00:14:01) Elon Musk’s xAI Unveils Grok-1.5 Vision, Beats OpenAI’s GPT-4V
- (00:17:55) Reka releases Reka Core, its multimodal language model to rival GPT-4 and Claude 3 Opus
- (00:21:50) Cohere Compass Private Beta: A New Multi-Aspect Embedding Model
- (00:23:48) Amazon Music’s Maestro lets listeners make AI playlists
- (00:24:36) Snap plans to add watermarks to images created with its AI-powered tools
Applications & Business
Projects & Open Source
- (00:44:08) OpenEQA: Embodied Question Answering in the Era of Foundation Models
- (00:50:03) Introducing Idefics2: A Powerful 8B Vision-Language Model for the community
Research & Advancements
- (00:51:21) RHO-1: Not All Tokens Are What You Need
- (00:57:21) Scaling Laws for Fine-Grained Mixture of Experts
- (01:03:20) Chinchilla Scaling: A replication attempt
- (01:07:18) China develops new light-based chiplet that could power artificial general intelligence — where AI is smarter than humans
- (01:10:45) OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments
Policy & Safety
Synthetic Media & Art
- (01:30:25) Medium bans AI-generated content from its paid Partner Program

Get the Snipd
podcast app

Unlock the knowledge in podcasts with the podcast player of the future.

AI-powered
podcast player

Listen to all your favourite podcasts with AI-powered features

Discover
highlights

Listen to the best highlights from the podcasts you love and dive into the full episode

Save any
moment

Hear something you like? Tap your headphones to save it with AI-generated key takeaways

Share
& Export

Send highlights to Twitter, WhatsApp or export them to Notion, Readwise & more

AI-powered
podcast player

Listen to all your favourite podcasts with AI-powered features

Discover
highlights

Listen to the best highlights from the podcasts you love and dive into the full episode

Last Week in AI

#163 - Llama 3, Grok-1.5 Vision, new Atlas robot, RHO-1, Medium ban

Podcast summary created with Snipd AI

Quick takeaways

Deep dives

Selective Language Modeling: Efficient Training on Useful Tokens

Optimal Granularity in Mixture of Experts

R-H-O-1: Training Dynamics in Language Models

Scaling Loss for Mixture of Experts

Study Reveals Hardware Costs of Fine-Grained Communication

Medium Implements Policy Against Fully AI-Generated Content

Optimizing Model Training with Compute Power and Data Size

Strategic Implications of Model Training Teasers

Get the Snipd
podcast app

AI-powered
podcast player

Discover
highlights

Save any
moment

Share
& Export

AI-powered
podcast player

Discover
highlights

Last Week in AI

#163 - Llama 3, Grok-1.5 Vision, new Atlas robot, RHO-1, Medium ban

Podcast summary created with Snipd AI

Quick takeaways

Deep dives

Selective Language Modeling: Efficient Training on Useful Tokens

Optimal Granularity in Mixture of Experts

R-H-O-1: Training Dynamics in Language Models

Scaling Loss for Mixture of Experts

Study Reveals Hardware Costs of Fine-Grained Communication

Medium Implements Policy Against Fully AI-Generated Content

Optimizing Model Training with Compute Power and Data Size

Strategic Implications of Model Training Teasers

Get the Snipdpodcast app

AI-poweredpodcast player

Discoverhighlights

Save anymoment

Share& Export

AI-poweredpodcast player

Discoverhighlights

Get the Snipd
podcast app

AI-powered
podcast player

Discover
highlights

Save any
moment

Share
& Export

AI-powered
podcast player

Discover
highlights