Product data science for Microsoft AI, data scientist's role of GenAI, how to deal with burn out - Sid Sharan - The Data Scientist Show #077
Jan 15, 2024
auto_awesome
Sid Sharan, Senior Data and Applied Scientist at Microsoft, discusses evaluating AI products, OpenAI API for sentiment analysis, data science team culture, collaboration between data scientists and product managers, dealing with burnout, and the role of product data scientists in the age of generative AI tools.
Microsoft evaluates AI products based on their potential value and integration into existing product suites to create value for end users.
Product data scientists at Microsoft play a crucial role in evaluating AI products built with large language models by understanding the value and integration potential, assessing the impact and revenue generation, and providing insights to product teams.
Data-driven decision-making is an integral part of Microsoft's culture, and product data scientists work closely with product managers to define key metrics for success and evaluate the impact on global metrics like monthly active users and engagement.
Deep dives
Determining commercially viable AI products
Microsoft evaluates AI products based on their potential value and integration into existing product suites to create value for end users. The focus is on adding value and making day-to-day tasks easier for users. The viability of a product is assessed based on the value it can provide, such as integration into Azure to enhance solutions or features like Co-pilot that lead to increased document completions and user engagement.
Evaluating AI products and the role of product data science
Product data scientists at Microsoft play a crucial role in evaluating AI products built with large language models. They help differentiate between viable AI products and mere 'bright shiny objects'. By understanding the value and integration potential, they assess the impact and revenue generation of AI products. Their responsibility is to understand what can be successfully integrated, bring value to end users, and provide insights to product teams.
The role of data-driven decisions in Microsoft and transitioning to data science
Data-driven decision-making is an integral part of Microsoft's culture. The guest speaker, Sid, transitioned from a business background to data science by learning SQL, Python, and machine learning. He initially started working on data-focused projects at his previous company and gradually developed the skills and expertise to become a product data scientist at Microsoft.
Determining the success and potential of AI products
To assess the success and potential of AI products at Microsoft, the focus is on evaluating key metrics and gathering user feedback. The product data scientists work closely with product managers to define key metrics for success and align product roadmaps accordingly. Actions taken by administrators, such as thread deletion or muting, are used as indicators of success. Additionally, evaluating the impact on global metrics like monthly active users and engagement is crucial in determining the viability and potential of AI products.
Challenges and decision-making in product data science
Challenges in product data science include ensuring accurate sentiment analysis, keeping up with evolving AI models, and maintaining trust among stakeholders. The use of open AI API for sentiment analysis requires multi-pass analysis and validation. Product data scientists must also evaluate the impact and potential risks of AI models, align with stakeholders, and make decisions based on comprehensive analysis rather than relying solely on statistical significance. Building trust and effective communication with product managers and engineers is essential in navigating these challenges.
Siddhartha Sharan is a Senior Data and Applied Scientist at Microsoft, helping product teams make data-driven decisions. Currently he is working on an AI product built with OpenAI APIs for sentiment analysis. We talked about how he evaluates AI products built with large language models at Microsoft, product data science, and how he went from a business background to data science. Subscribe to Daliana's newsletter on www.dalianaliu.com for more on data science and career.
(00:00:00) Introduction (00:05:20) How does Microsoft evaluate AI product (00:16:17) Using OpenAI API for sentiment analysis (00:25:29) Microsoft data science team culture (00:26:52) DS, PM collaboration (00:28:29) Three steps to build trust in data science (00:30:13) How did he got into Microsoft (00:34:09) Level up in Genetech (00:36:09) ML engineer vs Product DS (00:37:43) Core skills in product DS (00:40:20) Hiring (00:42:47) How to deal with burnout (00:45:03) Should you over work to earn trust? (00:45:44) Daliana's story about first day at Amazon (00:49:54) Will AI replace data scientists? (00:51:32) Data scientist's role of GenAI (00:54:32) How to keep up with GenAI
Get the Snipd podcast app
Unlock the knowledge in podcasts with the podcast player of the future.
AI-powered podcast player
Listen to all your favourite podcasts with AI-powered features
Discover highlights
Listen to the best highlights from the podcasts you love and dive into the full episode
Save any moment
Hear something you like? Tap your headphones to save it with AI-generated key takeaways
Share & Export
Send highlights to Twitter, WhatsApp or export them to Notion, Readwise & more
AI-powered podcast player
Listen to all your favourite podcasts with AI-powered features
Discover highlights
Listen to the best highlights from the podcasts you love and dive into the full episode