The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence)

OLMo: Everything You Need to Train an Open Source LLM with Akshita Bhagia - #674

Mar 4, 2024

32:12

Snipd AI

Akshita Bhagia discusses the OLMo language model with a unique open-source approach. The OLMo umbrella includes projects like Dolma and Paloma. The importance of open-training datasets and data curation filters are emphasized. The podcast explores dataset contamination, task specificity, and the evolution of training data transparency.

AI Summary

Highlights

AI Chapters

Episode notes

Podcast summary created with Snipd AI

Quick takeaways

OLMo provides transparent pre-training data and tools to foster collaborative research.

Dolma dataset offers three trillion tokens from public data for analyzing model capabilities across domains.

Deep dives

The Motivation Behind Almost Project

The Almost project aims to address the lack of transparency in language model development by providing truly open language models. This initiative was driven by the necessity for researchers to access complete details of model training data and pre-training specifics. By releasing the 1B and 7B versions of the models alongside comprehensive pre-training data, training code, logs, and evaluation tools, Almost seeks to foster collaborative research and avoid redundant, costly experiments.

Importance of Open-Training Data Sets for Research

01:21

Insights on Training Neural Networks

01:15

Utilizing Evaluation Signals in AI Model Training

01:12

Introduction

4min

Comparing OMO Model with Existing Models and Emphasizing Collaborative Research

2min

Discussion on the Importance of Open-Training Datasets and Dolma Release

2min

Data Collection and Curation for an Open Source LLM

2min

Exploring Dataset Contamination, Task Specificity, and Model Training for Language Models

2min

Exploring the Evolution of Training Data Transparency

20min

Today we’re joined by Akshita Bhagia, a senior research engineer at the Allen Institute for AI. Akshita joins us to discuss OLMo, a new open source language model with 7 billion and 1 billion variants, but with a key difference compared to similar models offered by Meta, Mistral, and others. Namely, the fact that AI2 has also published the dataset and key tools used to train the model. In our chat with Akshita, we dig into the OLMo models and the various projects falling under the OLMo umbrella, including Dolma, an open three-trillion-token corpus for language model pretraining, and Paloma, a benchmark and tooling for evaluating language model performance across a variety of domains.

The complete show notes for this episode can be found at twimlai.com/go/674.

Get the Snipd
podcast app

Unlock the knowledge in podcasts with the podcast player of the future.

AI-powered
podcast player

Listen to all your favourite podcasts with AI-powered features

Discover
highlights

Listen to the best highlights from the podcasts you love and dive into the full episode

Save any
moment

Hear something you like? Tap your headphones to save it with AI-generated key takeaways

Share
& Export

Send highlights to Twitter, WhatsApp or export them to Notion, Readwise & more

AI-powered
podcast player

Listen to all your favourite podcasts with AI-powered features

Discover
highlights

Listen to the best highlights from the podcasts you love and dive into the full episode

The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence)

OLMo: Everything You Need to Train an Open Source LLM with Akshita Bhagia - #674