OpenAI, AGI, LLMs Eval & Applied ML with Reah Miyara #47
May 16, 2024
auto_awesome
Reah Miyara, an expert in LLMs evaluation at OpenAI with a rich background at Google and IBM, shares his career journey from software engineering to product leadership. He discusses the evolution of AI, focusing on the importance of validating innovations in real-world applications. Reah delves into the complexities of LLM evaluation and the significance of safety metrics in AI models. He emphasizes the vital role of feedback in career growth and offers insights into the future landscape of generative AI and its implications for society.
Reah Miyara's transition from product management at Google to LLM evaluation at OpenAI highlights the necessity of integrating user-centric approaches with technical expertise in AI development.
The podcast emphasizes the critical role of safety metrics in LLM evaluation to ensure responsible use of AI technology and its potential to solve complex societal problems.
Deep dives
Journey into AI and Product Management
The guest's journey into artificial intelligence began at the University of California, Berkeley, where they started a club focused on building drones. This experience sparked an interest in AI, particularly how algorithms can emulate human decision-making. Their first internship at NASA's Jet Propulsion Laboratory deepened this interest, as they contributed to software for the Mars rover. Ultimately, this passion for problem-solving led them to transition from software engineering at IBM to a product management role, underscoring the blend of technical knowledge and user-centric thinking in product development.
Experience at Google and the Use of Graph Algorithms
The guest served as a product lead at Google AI, where they focused on the graph algorithms portfolio, impacting various applications like YouTube's recommendation system and autonomous driving technologies. This role emphasized both problem-oriented and innovative approaches to product management, allowing them to identify and address costly inefficiencies within existing processes. By collaborating with internal teams, they sought to validate novel algorithms against real customer problems, enhancing various Google products. The successful application of graph machine learning tools demonstrated the capacity for significant impact across multiple organizational efforts.
Role at OpenAI and LLM Evaluation
At OpenAI, the guest now works on the LLM evaluation team, tasked with measuring model capabilities and defining success in a rapidly evolving landscape. This involves continuously refining what constitutes a 'good' model, considering various performance metrics and requirements for intelligence across diverse domains. A notable challenge lies in effectively comparing candidate models, as improvements in one area may coincide with regressions in others. This nuanced evaluation requires integrating different metrics into actionable insights, ensuring that evaluation results translate into practical enhancements.
Future of Generative AI and Safety in Model Evaluation
Looking ahead, the guest expresses optimism for generative AI and the potential for AGI to resolve complex societal issues through innovative solutions. They underscore the importance of safety metrics in model evaluation, ensuring that the technology is used responsibly and does not facilitate harmful actions. Evaluation categories span various dimensions, including safety, performance, and user relevance, to create a comprehensive assessment framework. Ultimately, continuous feedback loops and robust validation processes are critical for advancing AI technologies while prioritizing ethical considerations.
Our guest today is Reah Miyara. Reah is currently working on LLMs evaluation at OpenAI and previously worked at Google and IBM.
In our conversation, Reah shares his experience working as a product lead for Google's graph-based machine learning portfolio. He then explains how he joined OpenAI and his role there. We finally talk about LLMs evaluation, AGI, LLMs safety and the future of the field.
If you enjoyed the episode, please leave a 5 star review and subscribe to the AI Stories Youtube channel.