Preparing Data to Measure True Machine Learning Model Performance

How do you prepare a dataset for machine learning (ML)? How do you go beyond cleaning the data and move toward measuring how the model performs? This week on the show, Jodie Burchell, developer advocate for data science at JetBrains, returns to talk about strategies for better ML model performance.

Jodie starts by defining some terms for the conversation. We talk about targets, features, and supervised learning.

We discuss three common ways that data can alter model performance and which Python tools can help spot and avoid them. Jodie shares personal experiences of working through these pitfalls. We also share a healthy collection of resources to explore and learn more.

Topics:

• 00:00:00 – Introduction
• 00:01:46 – Recent conference talks
• 00:03:24 – How to prepare your data for model performance
• 00:04:24 – Vocabulary: target, features, and supervised learning
• 00:06:28 – The curse of dimensionality
• 00:08:57 – Overfitting
• 00:11:08 – Underfitting
• 00:12:11 – Splitting the dataset
• 00:13:39 – K-fold cross validation
• 00:18:30 – Data leakage
• 00:21:36 – Checking for duplicates
• 00:26:23 – Applying transformations only after splitting data
• 00:31:16 – Imbalanced data
• 00:36:36 – Using ML to balance data
• 00:41:05 – Informing your model of the imbalance
• 00:42:56 – Video Course Spotlight
• 00:44:20 – Accuracy used as a measure
• 00:49:05 – Scikit-learn method classification_table
• 00:50:43 – Jet Brains blog post and conference talk
• 00:52:18 – How can people follow your work online?
• 00:54:39 – Upcoming webinars
• 00:56:20 – Thanks and goodbye

Show Links:

• How to Prepare Your Dataset for Machine Learning and Analysis - The JetBrains Datalore Blog
• Curse of dimensionality - Wikipedia
• Overfitting vs. Underfitting: A Complete Example - Will Koehrsen
• A Gentle Introduction to k-fold Cross-Validation - MachineLearningMastery.com
• sklearn.model_selection.train_test_split — scikit-learn documentation
• Cross-validation: evaluating estimator performance — scikit-learn documentation
• sklearn.model_selection.cross_val_score — scikit-learn documentation
• Data Leakage And Its Effect On The Performance of An ML Model
• pandas.DataFrame.duplicated — pandas documentation
• pandas GroupBy: Your Guide to Grouping Data in Python – Real Python
• pandas.DataFrame.groupby — pandas documentation
• Difference between fit(), transform() and fit_transform() method in Scikit-learn - Aishwarya Chand: Nerd For Tech
• Imbalanced Data in Machine Learning - Google Developers
• Under-sampling — imbalanced-learn.org
• Over-sampling — imbalanced-learn.org
• Learn - Getting Started with Gretel.ai
• Classification on imbalanced data: Class weights - TensorFlow Core
• Tour of Evaluation Metrics for Imbalanced Classification - MachineLearningMastery.com
• CloudBrew - A two-day conference by AZUG, the Belgium Microsoft Azure User Group
• Jodie Burchell’s Blog - Standard error
• Jodie Burchell 🇦🇺🇩🇪 (@t_redactyl) - Twitter
• Jodie Burchell 🇦🇺🇩🇪 (@t_redactyl@fosstodon.org) - Fosstodon
• JetBrains: Essential tools for software developers and teams

Level up your Python skills with our expert-led courses:

• Data Cleaning With pandas and NumPy
• Sneaky REST APIs With Django Ninja
• Combining Data in pandas With concat() and merge()

Support the podcast & join our community of Pythonistas