Evaluating 'Goodness' in AI

In episode #2 of Deployed: The AI Product Podcast, we meet with Mike Conover, co-founder & CEO of Brightwave to discuss the capabilities and challenges of building AI systems for financial research.
Brightwave is an AI research assistant for financial professionals. Their product generates insightful and trustworthy financial analyses on demand.
We get into the details of what it takes to make Brightwave work well, and lessons learned along the way including:

• Some of the limitations of LLMs, and what to do about them — especially when it comes to summarizing lots of content (tl;dr - long context windows don’t solve everything)
• How they’ve developed their eval suite through a practical iteration process among in-house finance experts, product, and engineering
• Thoughts on staffing AI engineering teams, including what he’s seen work to get strong software engineers up to speed working with LLMs

Let us know what you think in the comments.