Data Brew Season 1 Episode 5: Combining Machine Learning and MLflow with your Lakehouse
Jan 6, 2021
auto_awesome
The podcast discusses how Quby leverages ML to extract value from their data lake in the energy industry. They explore using energy data to create data-driven services and the challenges of clustering algorithms. They also discuss less intrusive monitoring methods, data transformation for privacy compliance, and obtaining permission from users.
36:00
AI Summary
AI Chapters
Episode notes
auto_awesome
Podcast summary created with Snipd AI
Quick takeaways
Quby leverages machine learning to extract additional value from their data lake and provides non-intrusive monitoring services to help users identify energy-saving opportunities.
QB has implemented MLflow to track models, parameters, and metrics, facilitating collaboration among team members and enabling efficient model comparison and iteration.
Deep dives
QB: Leveraging Machine Learning to Extract Value from Data Lake
Alyssa Versaput, a Machine Learning Engineer at QB, discusses how QB utilizes machine learning to extract additional value from their data lake. QB is a tech company based in the Netherlands that focuses on smart energy solutions. They gather IoT data from their smart thermostats deployed in households across Europe. QB helps users save energy by providing recommendations and updates for controlling heating and electricity usage efficiently. The company initially focused on hardware but transitioned into leveraging data using Apache Spark and Databricks. They started with proof of concepts and piloting services for a smaller user base, eventually scaling up to serve over 300,000 users. QB also addresses privacy concerns and ensures GDPR compliance. They prioritize user consent and engagement while providing non-intrusive monitoring services to help users identify energy-saving opportunities.
Monitoring and Alerting in QB's Machine Learning Models
QB has implemented monitoring and alerting systems for their machine learning models. They track data quantity, quality, and metrics throughout their pipelines. Daily trained models are logged, and metrics are monitored as time series. If the metrics deviate unexpectedly, alerts are triggered and displayed in Slack. QB is also working on analyzing and improving their alerting system, ensuring that the appropriate actions are taken when deviations occur. They have leveraged MLflow, a machine learning lifecycle management platform, which has allowed them to better track models, parameters, and metrics, facilitating collaboration among team members and enabling efficient model comparison and iteration.
Future Innovations: Use Cases and Features on the Horizon
QB is exploring future use cases and features in the smart energy domain. They are focused on assisting energy providers and end-users in preparing for the future, such as upgrading houses for climate change and introducing renewable energy technologies like solar panels and heat pumps. QB aims to engage users by providing them with insights and opportunities that are beneficial to their energy consumption. Additionally, they are conducting research on less intrusive monitoring techniques to observe user behavior and ensure energy efficiency without compromising privacy and surveillance concerns. QB places a strong emphasis on GDPR compliance and adheres to user privacy contracts.
Benefits of MLflow for QB's Machine Learning Workflows
The implementation of MLflow has brought several benefits to QB's machine learning workflows. It has standardized and streamlined model tracking, allowing for easier comparison and collaboration among team members. With MLflow, they can easily share experiment results, track model performance, and iterate on models more efficiently. MLflow also helps to reduce duplicate efforts and ensures fairness when evaluating and selecting models. QB appreciates how MLflow simplifies the process of comparing and assessing various models, enabling them to rapidly iterate and select the best models for deployment.
Ellissa Verseput, ML Engineer at Quby, joins Denny and Brooke to discuss how Quby leverages ML to extract additional value from their data lake and how they manage this process.
