The Data Engineering Show Vector Databases Won’t Replace SQL - Andy Pavlo
Jun 4, 2024
Andy Pavlo, an Associate Professor at Carnegie Mellon University, dives into the intricacies of database internals and optimization. He discusses the development of Autotune, a machine learning project aimed at improving database performance. Andy explains why SQL remains essential despite the allure of new technologies, like vector databases. He also highlights the synergy between academic research and practical applications, demonstrating how innovations in query optimization keep SQL relevant in a rapidly evolving landscape.
AI Snips
Chapters
Transcript
Episode notes
Use ML to Tune Database Knobs
- Tune database configuration knobs like caching policies and buffer sizes to optimize system performance.
- Machine learning models can predict behavior changes from these tuning actions to find the best configuration for workloads.
NoisePage Challenges During Pandemic
- Andy Pavlo took on many students during the pandemic to work on NoisePage, which scaled the project beyond sustainable limits.
- This led to code quality issues and contributed to the project's eventual pause.
Tune Databases Without Clones
- Database tuning often assumes a spare clone and workload trace to optimize without impacting production.
- Many customers can't afford or manage clones, so tuning must be done carefully on production systems.