Shreya Shankar, a PhD student at UC Berkeley, shares her insights on DocETL, a system designed for optimizing LLM-powered data processing pipelines. They discuss the fascinating challenges of intelligent data extraction from unstructured sources like PDFs and the pivotal role of human insight in prompt engineering. Shreya emphasizes the need for tailored benchmarks in data processing tasks and showcases real-world applications, including police misconduct data collection. The conversation highlights the balance between automation and human collaboration in AI systems.

About This EpisodeShreya Shankar is a computer scientist, PhD student in databases at UC Berkeley, and co-author of "Operationalizing Machine Learning: An Interview Study", an ethnographic interview study with 18 machine learning engineers across a variety of industries on their experience deploying and maintaining ML pipelines in production.Shreya explains the high-level findings of "Operationalizing Machine Learning"; variables that indicate a successful deployment (velocity, validation, and versioning), common pain points, and a grouping of the MLOps tool stack into four layers. Shreya and Lukas also discuss examples of data challenges in production, Jupyter Notebooks, and reproducibility.Show notes (transcript and links): http://wandb.me/gd-shreya---💬 *Host:* Lukas Biewald---*Subscribe and listen to Gradient Dissent today!*👉 Apple Podcasts: http://wandb.me/apple-podcasts​​👉 Google Podcasts: http://wandb.me/google-podcasts​👉 Spotify: http://wandb.me/spotify​

Join Shreya Shankar, a UC Berkeley researcher specializing in human-centered data management systems, as she navigates the exciting world of large language models (LLMs). Discover her insights on the shift from traditional machine learning to LLMs and the importance of data quality over algorithm issues. Shreya shares her innovative SPaDE framework for improving AI evaluations and emphasizes the need for human oversight in AI development. Plus, explore the future of low-code tools and the fascinating concept of 'Habsburg AI' in recursive processes.

In this episode, Shreya Shankar, a computer scientist pursuing her PhD in databases at UC Berkeley, discusses the challenges faced by machine learning engineers, including ensuring consistent performance and accuracy in production ML models. She also explores the importance of preserving information for anomaly detection, quantitative statistics for data cleaning, and the evolution from MLOps to LLMOps and FMOps. The speakers discuss the difficulties of implementing ML models and ML monitoring, as well as varying perspectives on AI as a product and its impact on customer expectations.

Experts from Amazon, Hex, Modal, Parlance Labs, and UC Berkeley share lessons learned from working with Large Language Models. They discuss the importance of evaluation and monitoring in LLM applications, data literacy in AI, the fine-tuning dilemma, real-world insights, and the evolving roles of data scientists and AI engineers.