Eye On A.I.

#187 Mohamed Elgendy: Systematic Testing for Generative AI Models with Kolena

May 18, 2024

Mohamed Elgendy from Kolena talks about systematic testing for AI models, emphasizing the importance of structured frameworks, human evaluation, and collaboration with industry bodies to shape AI standards. They discuss challenges in testing AI models, defining metrics, fine-tuning generative AI models, and risk mitigation strategies, highlighting the importance of ongoing monitoring and standardization.

57:36

Creator website

Episode guests

Mohamed Elgendy

AI Summary

AI Chapters

Episode notes

Podcast summary created with Snipd AI

Quick takeaways

Kolena aims to set new standards in AI testing by addressing non-deterministic nature of ML models with systematic validation framework.

Platform leverages millions of test runs monthly to establish comprehensive evaluation for diverse AI applications.

Colena plans to automate AI model testing with tailored evaluation models to enhance efficiency and accuracy.

Deep dives

Colena: A Platform for AI Quality Assurance

Colena is an AI quality platform that assists teams in testing, validating, and ensuring the performance of AI products. The platform aims to define a gold standard for AI evaluation by emphasizing the importance of systematic frameworks in AI quality assurance. By addressing the challenges of testing machine learning systems, Colena focuses on building a comprehensive solution for end-to-end model quality assessment. Through collaborations with MLOps communities, Colena strives to advance new standards and best practices for AI model testing and quality.

Introduction

5min

Challenges and Solutions in Testing AI Models

5min

Importance of Defining and Testing Metrics for AI Models

3min

Testing and Fine-Tuning Generative AI Models

29min

Risk Mitigation Strategies for Generative AI Models

14min

Closing Remarks, AI Impact, and Sponsors' Promotions

2min

This episode is sponsored by Oracle. AI is revolutionizing industries, but needs power without breaking the bank. Enter Oracle Cloud Infrastructure (OCI): the one-stop platform for all your AI needs, with 4-8x the bandwidth of other clouds. Train AI models faster and at half the cost. Be ahead like Uber and Cohere.

If you want to do more and spend less like Uber, 8x8, and Databricks Mosaic - take a free test drive of OCI at https://oracle.com/eyeonai

In this episode of the Eye on AI podcast, join us as we dive into the realm of AI quality assurance with Mohamed Elgendy, CEO and co-founder of Kolena.

Discover the intricacies of ensuring robust AI performance as Mohamed shares his journey from a software engineer to leading an AI quality platform. Kolena is setting new standards in AI testing, addressing the non-deterministic nature of machine learning models with a systematic framework for validation.

The conversation navigates the challenges of AI testing, the importance of structured quality assurance, and the collaboration with industry bodies to shape the future of AI standards. Mohamed elaborates on Kolena's innovative approach to both human and automated evaluations, providing comprehensive testing solutions tailored to diverse customer needs.

We also explore real-world use cases and the platform's future plans to make AI testing more accessible and reliable. Mohamed's insights illuminate the path toward achieving high-quality AI systems, ensuring they perform as intended across various applications.

Tune in to understand the advancements propelling AI quality forward and how Kolena is pioneering the way in this crucial aspect of AI development.

Don't forget to like, subscribe, and hit the notification bell for more insights into the technologies driving the AI revolution.

This episode is sponsored by Shopify. Shopify is a commerce platform that allows anyone to set up an online store and sell their products. Whether you’re selling online, on social media, or in person, Shopify has you covered on every base. With Shopify you can sell physical and digital products. You can sell services, memberships, ticketed events, rentals and even classes and lessons.

Stay Updated:

Craig Smith Twitter: https://twitter.com/craigss

Eye on A.I. Twitter: https://twitter.com/EyeOn_AI