Guest:
Ellen Brandenburger – Product leader and coach; former head of product at Chegg Skills and Stack Overflow’s data licensing team.
What we cover in this episode:
- How Ellen joined Stack Overflow just two weeks before ChatGPT launched, reshaping the company’s future overnight
- The creation of Overflow AI: a team tasked with exploring “what’s just now possible” for developers
- Four iterations of conversational search:
- V1: a chat UI on top of keyword search
- V2: semantic search to handle natural questions
- V3: fallback to GPT-4 for gaps in Stack Overflow’s corpus
- V4: adding RAG for attribution and transparency
- Why attribution and transparency were critical for developer trust
- How the team used simple spreadsheets and subject-matter experts to evaluate answer accuracy, relevance, and completeness
- Why Stack decided to sunset conversational search despite heavy investment—what they learned and why it wasn’t wasted
- The pivot to data licensing: how Stack Overflow leveraged its 14M+ Q&A corpus to power LLM training and benchmarks
- Building industry benchmarks with subject-matter experts to prove Stack data improved LLM accuracy and relevance
Key lessons:
- Take one bite of the apple at a time—prototype, learn, iterate
- Product in the AI era means managing probabilities, not certainties
Links & References:
