The AI Skill That Will Define Your PM Career in 2025 | Aman Khan (Arize)
Jan 12, 2025
auto_awesome
Aman Khan, Director of Product at Arize AI, discusses essential AI evaluation skills for product managers. He emphasizes the necessity of empathy in understanding user needs and the importance of curiosity in product development. Aman outlines five critical skills to cultivate now, explaining how anyone can engage with AI, regardless of their title. He highlights the value of performance evaluations in customer support AI, advocating for the synergy of human oversight and data. The conversation wraps up with a call to embrace the AI revolution.
Empathy for users is essential for PMs to effectively navigate AI development and articulate user experiences accurately.
Building curiosity and proficiency in AI fundamentals, along with rapid prototyping, enables PMs to drive innovative product solutions.
Writing structured evaluations is crucial for PMs to assess AI performance against user expectations and improve product iterations.
Deep dives
Empathy in Product Management
Developing empathy for users is a crucial skill for product managers (PMs) working on AI systems. By focusing on understanding user needs, PMs are encouraged to articulate user experiences clearly in written form. This process often involves modeling human behavior, which can be subjective and non-deterministic, making it challenging to predict outcomes. As a result, the iterative nature of AI development emphasizes the importance of fine-tuning product iterations based on user feedback and data analysis.
Key Skills for AI Product Managers
Aspiring AI PMs need to build a set of key skills to navigate the evolving landscape of artificial intelligence. Fundamental knowledge of AI and machine learning, along with a deep understanding of customer needs, are essential components. Curiosity and the ability to prototype solutions will drive innovation within AI-based products. Lastly, effectively evaluating and measuring system performance through structured evaluations will differentiate successful AI PMs from their peers.
Eval Writing as a Core Competency
Writing effective evaluations, or 'evals', is increasingly recognized as a vital skill for PMs working with AI. Eval writing requires PMs to immerse themselves in the user's perspective and assess whether AI outputs align with user expectations. This process, however, can be complicated by variability in human inputs and subjective interpretations of AI performance. As AI systems become more complex, developing standardized criteria for evaluations can help manage these challenges more effectively.
Prototyping with AI Tools
Utilizing AI tools for rapid prototyping can significantly enhance a PM's ability to iterate on product features. By leveraging tools that automatically generate code based on specific requirements, PMs can quickly test ideas and gather user feedback. For example, platforms like Replit can facilitate the development of functional prototypes during brainstorming sessions. This agile approach enables teams to refine concepts in real-time, allowing for quicker adaptation to user needs.
Building Better AI Systems
To improve AI products effectively, organizations must employ a structured approach to evaluation and development. This includes gathering real-world data, systematically assessing AI outputs against user scenarios, and continuously iterating based on findings. By breaking down the system into manageable components, PMs can identify areas for improvement and measure overall effectiveness. The integration of user feedback and performance metrics will ultimately lead to the design of more reliable and user-friendly AI systems.
Aman is the Director of Product at Arize AI. Last year, we both heard the Chief Product Officers of OpenAI and Anthropic share that AI evaluations will be the most important skill for PMs in 2025. Aman gave me a crash course on how to build this critical skill in our interview.
Timestamps:
(00:00) Evals force you to get into your user's shoes
(03:42) 5 skills to build right now to become a great AI PM
(07:43) How curiosity leads to better AI products
(10:20) You can build AI without the job title
(17:00) A deep dive into why AI evals are so important
(24:53) Example of running evals for an AI customer support agent
(31:32) When humans need to be in the loop for evals
(35:46) How to get better at writing evals right now
(39:39) My personal prompt and eval for transcripts
(44:23) Catching the AI wave is alot like surfing
Get the takeaways: https://creatoreconomy.so/p/the-ai-skill-that-will-define-your-pm-career-aman-khan