Test & Code

Data science, data engineering, data analysis, and machine learning are part of the recent massive growth of Python. But really what is data science? Vicki Boykis helps me understand questions like: No really, what is data science? What does a data pipeline look like? What is it like to do data science, data analysis, data engineering? Can you do analysis on a laptop? How big does data have to be to be considered big? What are the challenges in data science? Does it make sense for software engineers to learn data engineering, data science, pipelines, etc? How could someone start learning data science? Also covered: A type work (analysis) vs B type work (building) data lakes and data swamps predictive models data cleaning development vs experimentation Jupyter Notebooks Kaggle ETL pipelines I learned a lot about the broad field of data science from talking with Vicki.Special Guest: Vicki Boykis.Links:How to Lie with Statistics : Darrell HuffShould you replace Hadoop with your laptop?KaggleProject JupyterSoviet Art Bot — A bot that finds socialist realism paintings and tweets them out

Michael Kennedy of Talk Python and Python Bytes fame joins Brian to talk about being a great guest and what to expect. Even if you have never wanted to be on a podcast, you might learn some great tips. A few of the things we talk about will be helpful for other endeavors, like public speaking, guest blog posts, look for unsolicited job opportunities. Some people have never been on a podcast before, and are possibly freaked out about some of the unknowns of being on a podcast. That's why we did this episode. Michael and I discuss a bunch of the niggly details so that you can be relaxed and know what to expect. Topics include: If you want to be on a podcast How to stand out and be someone a podcast would want to have on a show. How to suggest yourself as a guest and the topic you want to discuss. Picking a topic for a podcast What to do before the show to prepare Helping the host out with some information Some hardware (not much) Some software (all free) Sending info like bio, headshot, links, etc. What to expect the host or show to do before the recording. Where to record Sketching out some show topics with the host, maybe on a shared document. What to expect and do Right before the show During the conversation After the recording When it goes live (help promote it) Special Guest: Michael Kennedy.

What happens when 100% test code coverage just isn't enough. In this episode, we talk with Mahmoud Hashemi about glom, a very cool project in itself, but a project that needs more coverage than 100%. This problem affects lots of projects that use higher level programming constructs, like domain specific languages (DSLs), sub languages mini languages, compilers, and db query languages. Also covered: awesome Python applications versioning: 0-ver vs calver vs semver Special Guest: Mahmoud Hashemi.Links:Announcing glom — Restructured Data for PythonDomain-specific language - Wikipediaawesome-python-applications — Free software that works great, and also happens to be open-source Python.Meld — a visual diff and merge tool targeted at developers.ZeroVer: 0-based Versioning SemVer: Semantic Versioning CalVer: Calendar Versioningepisode 27: unit, integration, and system testing - Mahmoud Hashemi

Paul talks about the beginning years of Python. Talking about Python's beginnings is also talking about the Python community beginnings. Yes, it's reminiscing, but it's fun.Special Guest: Paul Everitt.Links:Python 1994 Panel Discussion PyCon 2017"Python 1994", PyBay2017Spam, Spam, Spam, ...

Luc Perkins joins the show to talk about "Seven Databases in Seven Weeks: A guide to modern databases and the NoSQL movement." We discuss a bit about each database: Redis, Neo4J, CouchDB, MongoDB, HBase, Postgres, and DynamoDB.Special Guest: Luc Perkins.Links:Seven Databases in Seven Weeks, Second Edition: A Guide to Modern Databases and the NoSQL MovementPostgreSQLRedisNeo4j Graph DatabaseCouchDBMongoDBHBaseDynamoDB

Brett Cannon discusses the changes afoot in Python packaging as a result of PEP 517, PEP 518, starting with "How did we get here?" and "Where are we going?" Discussed: flit Poetry tox Continuous Integration setup.py, MANIFEST.in, etc. pipenv what's with lock files applications (doesn't go on PyPI) vs libraries (goes on PyPI) workflows dependency resolution deployment dependencies vs development dependencies will lock files be standarized multiple lock files requirements.txt Special Guest: Brett Cannon.Links:FlitPoetryPython Bytes #100 : The big 100 with special guests PEP 517 -- A build-system independent format for source trees | Python.orgPEP 518 -- Specifying Minimum Build System Requirements for Python Projects | Python.org

Andy Knight joins me in discussing the concept of feature testing. A feature tests is "a test verifying a service or library as the customer would use it, but within a single process." That was a quote from an article that appeared on the Twitter engineering blog. The article describes a shift away from class tests towards feature tests, the benefits of the shift, and some reactions to it. Feature tests are similar to something I used to call "functional subcutaneous integration test", but it's a way better name, and I plan to use it more often. The idea fits well with my testing philosophy. Andy Knight is someone still holding onto the testing pyramid. So I thought it would be fun to ask him to discuss feature testing with me. I think it's a balanced discussion. I hope you enjoy it and learn something.Special Guest: Andy Knight.Links:Twitter engineering blog article describing Feature Testing : The testing renaissance

Anthony Shaw joins Brian to discuss flaky tests and flaky test suites. What are flaky tests? Is it the same as fragile tests? Why are they bad? How do we deal with them? What causes flakiness? How can we fix them? How can we avoid them? Proactively rooting out flakiness Test design GUI tests Sharing solutions Special Guest: Anthony Shaw.Links:Dropbox article on flaky testsMicrosoft article on flaky testspytest-rerunfailures: a py.test plugin that re-runs failed tests up to -n times to eliminate flakey failurespytest-randomly: Pytest plugin to randomly order tests and control random.seedpytest-random-order: pytest plugin to randomise the order of tests with some control over the randomnessmath.isclose()numpy.isclose()pytest.approx() — approxAnthony's testing article on RealPythonGhost Inspectorwily: A Python application for tracking, reporting on timing and complexity in tests

tox is a simple yet powerful tool that is used by many Python projects. tox is not just a tool to help you test a Python project against multiple versions of Python. In this interview, Oliver and Brian just scratch the surface of this simple yet powerful automation tool. This is from the tox documentation: tox is a generic virtualenv management and test command line tool you can use for: checking your package installs correctly with different Python versions and interpreters running your tests in each of the environments, configuring your test tool of choice acting as a frontend to Continuous Integration servers, greatly reducing boilerplate and merging CI and shell-based testing. Yet tox is so much more. It can help create development environments, hold all of your admin scripts, ... I hope you enjoy this wonderful discussion of tox with Oliver Bestwalter, one of the core maintainers of tox.Special Guest: Oliver Bestwalter.Links:tox project documentationtox recreate : "Have you turned it off and on again?" for tox"Hello world" of toxtox also has pluginstalk by Bernát Gábor about a tox based workflow at EuroPython 2018adding a description to your environmentsdetox - distributed toxdevpi: private package indexPyCharm plugin to easily set the project interpreter via context menu: PyVenvmanagepower mode in atomPower Mode for PyCharm

The story of how I came to find a good user interface for running and debugging automated tests is interleaved with a multi-year effort of mine to have a test workflow that’s works smoothly with product development and actually speeds things up. It’s also interleaved with the origins of the blog pythontesting.net, this podcast, and the pytest book I wrote with Pragmatic. It’s not a long story. And it has a happy ending. Well. It’s not over. But I’m happy with where we are now. I’m also hoping that this tale of my dedication to, or obsession with, quality and developer efficiency helps you in your own efforts to make your daily workflow better and to extend that to try to increase the efficiency of those you work with.Links:pythontesting.net