Ep 22: Notion AI Engineer Linus Lee: Behind the Scenes of Notion AI
Nov 16, 2023
auto_awesome
Linus Lee, an AI engineer at Notion, shares behind-the-scenes stories of one of the most widely used AI tools today. Topics include developing AI products, staffing, educating users, challenges in building Notion Q&A, partnerships with Anthropic and Open AI, iterating on interfaces, and over-hyped/under-hyped AI features.
Notion AI team focuses on finding the right abstractions for AI features, balancing flexibility and guidance for users.
Alternative architectures to transformers are considered underhyped and can lead to innovative solutions.
Iterating quickly and closely interacting with users during the ideation phase is crucial for building successful AI features.
Deep dives
Building AI Features at Notion
Notion AI team has successfully shipped AI features like AI writer and Autofill. They started with prototyping and quickly iterating on the output, dogfooding the features internally and gathering feedback. The team found that users mostly utilize pre-built prompts for popular use cases like summarization and improving writing, but also iterate on the AI outputs for their specific needs. The team focuses on finding the right abstractions for AI features, balancing flexibility and guidance for users. They are still exploring the building blocks and organization for Notion AI and considering partnerships and integration with other teams.
Challenges and Hype in AI
Context length is considered overhyped, as many tasks do not require extremely long sequences. The focus should be on retrieving relevant information and filtering noise. On the other hand, alternative architectures to transformers are considered underhyped. While transformers are efficient and effective for modeling long sequences, there may be other architectures better suited for the task. General approaches have been surprisingly effective in solving various tasks, and building general models for multiple tasks yields better understanding of the domain. Prototyping and experimenting with generality in AI interfaces can lead to innovative solutions.
Insights and Lessons Learned from Building Notion AI
Notion AI team found that iterating quickly and closely interacting with users during the ideation phase is crucial. They strike a balance between exploring possibilities and user problems. Evaluation and prompt engineering are challenging, but crucial for achieving quality outputs. The team has learned that models can transfer well across languages, and the right abstractions and building blocks enable customization and flexibility. Alternative architectures and a focus on generality in AI hold promise for future advancements.
The Challenges of Building AI Features for Notion
Building AI features for Notion presents several challenges. One key challenge is the diverse range of questions users ask, often poking at the edges of the model's capabilities. Anticipating these varied queries and constructing high-quality evaluations to address them proves difficult. The team also faces operational questions regarding customer needs, privacy, and security. Additionally, they grapple with deciding on the ideal scale and infrastructure. Despite exploring open source models, most of the tools used for Notion AI are built in-house to ensure agility and customization.
Evaluating Models, Custom Data Structures, and Iterative Development
Notion AI employs a spectrum of evaluation approaches. They use deterministic programmatic evaluations, human annotators, and engineers examining model outputs to understand their performance and improve it. Notion's custom data structures are designed to represent the complexities of notion documents effectively. By owning their tools, Notion can iterate quickly and tailor their solutions to their specific needs. While considering using off-the-shelf models, Notion's focus is understanding their tasks and developing evaluations and datasets aligned with those tasks.
Linus Lee is an AI engineer at Notion, one of the earliest and most effective adopters of AI. In the episode, Linus shares how Notion developed its AI products, including Writer, Autofill, and Q&A, which just launched on Tuesday. It was fascinating to learn how Notion structures its AI team and dogfoods its development process. Linus also explores the hardest to anticipate when going to market with new AI features, and how Notion thinks of its LLM partnerships. Overall, a wide-ranging conversation about the behind-the-scenes stories of one of the most widely used AI tools today.
(0:00) intro
(0:37) T-Swift
(2:07) Notion AI
(9:08) approach to staffing
(16:51) educating users and user behavior
(22:32) challenges in developing Notion Q&A
(30:42) working with Anthropic and Open AI
(35:50) avoiding hallucinations
(36:23) switching AI models
(39:32) iterating on interfaces
(42:03) over-hyped/under-hyped
(48:03) Midjourney
(51:07) Pat and Jacob debrief
With your co-hosts:
@ericabrescia
- Former COO Github, Founder Bitnami (acq’d by VMWare)
@patrickachase
- Partner at Redpoint, Former ML Engineer LinkedIn
@jacobeffron
- Partner at Redpoint, Former PM Flatiron Health
@jordan_segall
- Partner at Redpoint
Remember Everything You Learn from Podcasts
Save insights instantly, chat with episodes, and build lasting knowledge - all powered by AI.
