The Importance of Attention

On this episode, we’re joined by Aidan Gomez, Co-Founder and CEO at Cohere. Cohere develops and releases a range of innovative AI-powered tools and solutions for a variety of NLP use cases.

We discuss:

- What “attention” means in the context of ML.

- Aidan’s role in the “Attention Is All You Need” paper.

- What state-space models (SSMs) are, and how they could be an alternative to transformers. 

- What it means for an ML architecture to saturate compute.

- Details around data constraints for when LLMs scale.

- Challenges of measuring LLM performance.

- How Cohere is positioned within the LLM development space.

- Insights around scaling down an LLM into a more domain-specific one.

- Concerns around synthetic content and AI changing public discourse.

- The importance of raising money at healthy milestones for AI development.

Aidan Gomez - https://www.linkedin.com/in/aidangomez/

Cohere - https://www.linkedin.com/company/cohere-ai/

Thanks for listening to the Gradient Dissent podcast, brought to you by Weights & Biases. If you enjoyed this episode, please leave a review to help get the word out about the show. And be sure to subscribe so you never miss another insightful conversation.

Resources:

- https://cohere.ai/

- “Attention Is All You Need”

#OCR #DeepLearning #AI #Modeling #ML