The Retort AI Podcast cover image

The Retort AI Podcast

Non-profits need to be businesses too

Jan 19, 2024
This podcast discusses the challenges of ML compute and evaluation, including AI2's transition to industry, the need for minimum viable resources, and government trust for non-profits. It examines the debate on a regulatory body for algorithms, exploring clear communication standards and evaluation tasks. It also explores the evolution of AI, uncomfortable choices, and the need for new approaches to add truth to public discourse.
37:10

Podcast summary created with Snipd AI

Quick takeaways

  • AI2 aims to transition from academic to nonprofit by establishing themselves as a trusted place for multi-organizational evaluation, requiring substantial compute power through the acquisition of GPUs.
  • NIST's role as a standard-setting organization can facilitate third-party assessment of AI systems, addressing concerns about cheating and contamination in evaluations, and ensuring responsible and safe development and deployment of AI systems.

Deep dives

Focus on AI2's transition to a more academic organization

AI2 is in the process of transitioning from being solely academic to proving themselves as a unique place for multi-organizational evaluation to happen. They aim to compare models and gain trust from the government as a nonprofit organization focused on understanding language models scientifically. This ambitious goal requires a significant amount of compute power, making the acquisition of GPUs crucial. While there are discussions on which institutions can manage GPUs better, AI2 emphasizes the need for clear communication and standards in the evaluation space, particularly for open-ended evaluations.

Get the Snipd
podcast app

Unlock the knowledge in podcasts with the podcast player of the future.
App store bannerPlay store banner

AI-powered
podcast player

Listen to all your favourite podcasts with AI-powered features

Discover
highlights

Listen to the best highlights from the podcasts you love and dive into the full episode

Save any
moment

Hear something you like? Tap your headphones to save it with AI-generated key takeaways

Share
& Export

Send highlights to Twitter, WhatsApp or export them to Notion, Readwise & more

AI-powered
podcast player

Listen to all your favourite podcasts with AI-powered features

Discover
highlights

Listen to the best highlights from the podcasts you love and dive into the full episode