Contracts and Code: The Realities of AI Development

12 snips

Sep 23, 2025

Valentino and Joe dive into the reality behind AI salaries, discussing the disparity between hype and actual compensation. They debate whether any company can truly dominate the LLM market, emphasizing incremental improvements over clear winners. The hosts explore the complexities of benchmarking AI models and the necessity for customized evaluation tools. New OpenAI features that enhance prompt engineering are discussed, alongside the balance between playful experimentation and standardization in Ruby, highlighting its role in AI development.

Ask episode

AI Snips

Chapters

Transcript

Episode notes

INSIGHT

Hyped Salaries Reflect Positioning

AI salary disparities are driven more by market positioning, scarcity, and equity than absolute individual worth.
Winning the AI race is unclear because improvements are incremental and competitors quickly catch up.

ADVICE

Benchmark Small, High-Value Targets

Start benchmark testing on targeted high-risk areas, not entire large codebases.
Focus on high complexity, high dependencies, and high FLOG score files to save cost and time.

INSIGHT

Prompts Behave Like Compiled Code

Every LLM has different prompting quirks, so identical prompts yield different outcomes across models.
Treat prompts like compiled source that may need reworking per model for best results.

Get the Snipd Podcast app to discover more snips from this episode

Get the app

In this episode, Valentino Stoll and Joe Leo unpack the widening gap between headline-grabbing AI salaries and the day-to-day realities of building sustainable AI products. From sports-style contracts stuffed with equity to the true cost of running large models, they explore why incremental gains often matter more than hype. The conversation dives into the messy art of benchmarking LLMs, the fresh evaluation tools emerging in the Ruby ecosystem, and new OpenAI features that change how prompts, tools, and reasoning tokens are handled. Along the way, they weigh the business math of switching models, debate standardisation versus playful experimentation in Ruby, and highlight frameworks like RubyLLM, Phoenix, and Leva that are reshaping how developers ship AI features.

Takeaways

The importance of marketing oneself in the tech industry.
Disparity in AI salaries reflects market demand and hype.
AI contracts often include equity, complicating true value assessment.
The AI race lacks clear winners, with incremental improvements across models.
User experience often outweighs model efficacy in AI products.
Prompt engineering is crucial for optimizing model performance.
Benchmarking AI models is complex and requires tailored evaluation sets.
Existing tools for AI evaluation are often insufficient for specific needs.
Cost analysis is critical when choosing AI models for business.
Incremental improvements in AI models may not meet user expectations. You can constrain tool outputs to specific grammars for flexibility.
Asking models to think out loud can enhance tool calls.
Reasoning tokens can be reused in subsequent AI calls.
Evaluating AI frameworks is crucial for business decisions.
Ruby's integration in AI is becoming more prominent.
The AI landscape is rapidly evolving, requiring adaptability.
Hype cycles can mislead developers about tool longevity.
Ruby offers a unique user experience for developers.
Tinkering with code fosters creativity and innovation.
The playful nature of Ruby can lead to unexpected insights.