The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence) Scaling Jupyter Notebooks with Luciano Resende - TWiML Talk #261
May 6, 2019
Luciano Resende, an Open Source AI Platform Architect at IBM, dives into the world of Jupyter Notebooks and scaling challenges. He discusses the evolution of JupyterHub and Enterprise Gateway, emphasizing their roles in data science and multi-user environments. The conversation touches on enhancing collaboration through better integration with Git and shared file systems. Luciano shares success stories from IBM Watson Studio and PayPal, and highlights exciting developments planned for the next version of Enterprise Gateway, focusing on Kubernetes optimization.
AI Snips
Chapters
Transcript
Episode notes
Scaling Jupyter at Financial Institutions
- Financial institutions needed to scale their Jupyter Notebook platforms.
- They had 500 data scientists needing to launch notebooks and crunch data concurrently.
Jupyter's Limitations
- Jupyter Notebooks evolved from Python consoles, inheriting limitations like single-user environments.
- Local kernel execution doesn't scale for enterprise needs, prompting add-on projects.
Complementary Tools
- JupyterHub and Jupyter Enterprise Gateway are not alternatives, but complementary tools.
- They work together to manage multi-user environments and scale backend kernel execution.