The Real Python Podcast Docker + Python for Data Science and Machine Learning
55:33
Docker Ensures Reproducibility
- Docker helps create isolated, reproducible environments for data science projects.
- It ensures collaboration without dependency conflicts across different setups.
Containers vs Virtual Machines
- Containers virtualize the OS, making them lighter and more portable than virtual machines.
- This allows easier sharing of reproducible environments without heavy hardware emulation.
Docker Best Practices
- Always pin versions of your dependencies and base container images for consistent builds.
- Use multi-stage builds to optimize image size and separate R&D from production environments.
Get the Snipd Podcast app to discover more snips from this episode
Get the app 1 chevron_right 2 chevron_right 3 chevron_right 4 chevron_right 5 chevron_right 6 chevron_right 7 chevron_right 8 chevron_right 9 chevron_right 10 chevron_right 11 chevron_right 12 chevron_right 13 chevron_right 14 chevron_right 15 chevron_right 16 chevron_right 17 chevron_right 18 chevron_right 19 chevron_right 20 chevron_right 21 chevron_right 22 chevron_right 23 chevron_right
Introduction
00:00 • 2min
Developer Advocates - What's Your Focus?
01:36 • 2min
Docker and Python for Machine Learning and Data Science
03:14 • 2min
Containers vs Virtual Machines
05:38 • 2min
Containers Are a Subset of Virtual Machines
07:33 • 2min
Reproducibility in Machine Learning
09:24 • 2min
The Challenges of Using Docker for Machine Learning
11:08 • 4min
Using Multi Stage Image Building for R&D Staging Development Environments
15:11 • 3min
The Advantages of Multi Stage Buildings
17:52 • 3min
Containerized Containerization
20:50 • 2min
Kubernetes
22:44 • 2min
Adian, What Are the Continuous Stages?
24:33 • 3min
Docker Containers for Machine Learning
27:05 • 2min
Do You Use Local or Cloud Computing for Data Science?
28:54 • 2min
Is There a Way to Do a Lot of NLP Analysis?
31:01 • 2min
What Are the Best Practices for Data Intensive Applications?
33:16 • 6min
Are There Any Good Resources to Get Started in Machine Learning?
39:02 • 4min
Pike in USEC - Mentoring Sprints
42:53 • 3min
The Cat Is Out of the Bag Now!
45:37 • 2min
The Power of Open Story
47:29 • 2min
What Are Your Other Hobbies?
49:09 • 2min
How to Optimize Python Code?
51:32 • 2min
What Are You Excited About Right Now in the World of Python?
53:52 • 2min
Docker is a common tool for Python developers creating and deploying applications, but what do you need to know if you want to use Docker for data science and machine learning? What are the best practices if you want to start using containers for your scientific projects? This week we have Tania Allard on the show. She is a Sr. Developer Advocate at Microsoft focusing on Machine Learning, scientific computing, research and open source.
Tania has created a talk for the PyCon US 2020 which is now online. The talk is titled “Docker and Python: Making them Play Nicely and Securely for Data Science and ML.” Her talk draws on her expertise in the improvement of processes, reproducibility and transparency in research and data science. We discuss a variety of tools for making your containers more secure and results reproducible.
Tania is passionate about mentoring, open-source, and its community. She is an organizer for Mentored Sprints for Diverse Beginners, and she talks about the upcoming online sprints for PyCon US 2020. We also discuss her plans to start a podcast.
Topics:
- 00:00:00 – Introduction
- 00:01:43 – Microsoft Senior Developer Advocate Role
- 00:04:07 – PyCon 2020 Talk - Docker and Python: making them play nicely
- 00:05:34 – What is Docker?
- 00:10:08 – Reproducibility of project results
- 00:12:03 – What are the challenges of using Docker for machine learning?
- 00:15:06 – Getting started suggestions
- 00:16:26 – What metadata should be included?
- 00:17:48 – Creating images through stages
- 00:21:16 – What about your data?
- 00:22:40 – Kubernetes: Orchestrating containers
- 00:24:37 – Continuing stages into testing
- 00:25:37 – What are tools for testing security?
- 00:27:07 – Challenges in using containers for ML
- 00:28:52 – What types of databases?
- 00:29:39 – Are you doing initial research on a local machine?
- 00:30:59 – An example of a recent ML project
- 00:32:16 – Papermill: parameterizing and executing notebooks
- 00:33:16 – NLP: Natural Language Processing
- 00:33:58 – Kaggle: Help us better understand COVID-19
- 00:34:42 – What are other best practices for data intensive projects?
- 00:39:13 – Resources to get started in machine learning?
- 00:40:30 – Mentored Sprints for Diverse Beginners
- 00:45:34 – Tania’s upcoming podcast
- 00:48:38 – A visiting fellow at the Alan Turing Institute
- 00:49:08 – Weight lifting
- 00:50:16 – Craft beer
- 00:52:09 – What is something you thought you knew in Python but were wrong about?
- 00:53:50 – What are excited about in the world of Python?
- 00:54:42 – Thank you and Goodbye
Show links:
- Tania Allard: Personal site
- Docker and Python: making them play nicely and securely for Data Science and ML - Tania Allard
- Slides for Docker and Python Talk
- Docker
- XKCD: Python Superfund Site
- Best practices for writing Dockerfiles
- Run Python Versions in Docker: How to Try the Latest Python Release
- Kubernetes: Production-Grade Container Orchestration
- Snyk: Securing open source and containers
- papermill: A tool for parameterizing and executing Jupyter Notebooks
- Natural Language Processing: Wikipedia article
- Natural Language Processing With spaCy in Python: Real Python article
- Kaggle: Help us better understand COVID-19
- datree.io: Scale Engineering organization
- repo2docker: Build, Run, and Push Docker Images from Source Code Repositories
- Jupyter Docker Stacks: A set of ready-to-run Docker images
- binder: Turn a Git Repo into a Collection of Interactive Notebooks
- Hands-On Machine Learning with Scikit-Learn and TensorFlow: O’Reilly
- Data Science from Scratch: O’Reilly
- Python for Data Analysis: Wes McKinney - Creator of Pandas
- Mentored Sprints for Diverse Beginners
- The Alan Turing Institute
- Easy Data Processing With Azure Fun - Tania Allard - PyCon 2020
- PEP 581 – Using GitHub Issues for CPython
- Python’s migration to GitHub - Request for Project Manager Resumes
Level up your Python skills with our expert-led courses: