How Do You Benchmark New Models at Gusto?

In this episode, Valentino Stoll and Joe Leo unpack the widening gap between headline-grabbing AI salaries and the day-to-day realities of building sustainable AI products. From sports-style contracts stuffed with equity to the true cost of running large models, they explore why incremental gains often matter more than hype. The conversation dives into the messy art of benchmarking LLMs, the fresh evaluation tools emerging in the Ruby ecosystem, and new OpenAI features that change how prompts, tools, and reasoning tokens are handled. Along the way, they weigh the business math of switching models, debate standardisation versus playful experimentation in Ruby, and highlight frameworks like RubyLLM, Phoenix, and Leva that are reshaping how developers ship AI features.

Takeaways

• The importance of marketing oneself in the tech industry.
• Disparity in AI salaries reflects market demand and hype.
• AI contracts often include equity, complicating true value assessment.
• The AI race lacks clear winners, with incremental improvements across models.
• User experience often outweighs model efficacy in AI products.
• Prompt engineering is crucial for optimizing model performance.
• Benchmarking AI models is complex and requires tailored evaluation sets.
• Existing tools for AI evaluation are often insufficient for specific needs.
• Cost analysis is critical when choosing AI models for business.
• Incremental improvements in AI models may not meet user expectations. You can constrain tool outputs to specific grammars for flexibility.
• Asking models to think out loud can enhance tool calls.
• Reasoning tokens can be reused in subsequent AI calls.
• Evaluating AI frameworks is crucial for business decisions.
• Ruby's integration in AI is becoming more prominent.
• The AI landscape is rapidly evolving, requiring adaptability.
• Hype cycles can mislead developers about tool longevity.
• Ruby offers a unique user experience for developers.
• Tinkering with code fosters creativity and innovation.
• The playful nature of Ruby can lead to unexpected insights.