218: Balancing test coverage with test costs - Nicole Tietz-Sokolskaya

Nicole is a software engineer and writer, and recently wrote about the trade-offs we make when deciding which tests to write and how much testing is enough.

We talk about:

• Balancing schedule vs testing
• How much testing is the right about of testing
• Should code coverage be measured and tracked
• Good refactoring can reduce code coverage
• Is it worth testing error conditions?
• Are rare error codes ok to just monitor?
• API drift and autospec
• Mitigating risk
• Deciding what to test and what not to test
• Focus testing on key money-making features 
• If there's a bug in this part of the code, how much business impact is there?
• Performance testing needs to approximately match real world workloads
• Cost of a service breaking vs the cost of creating, maintaining, and running tests
• Keeping test suites quick to minimize getting distracted

Links:

• Too much of a good thing: the trade-off we make with tests 
• Load testing is hard, and the tools are... not great. But why?
• Yet Another Rust Resource (YARR!)
• Goodhart's law - "When a measure becomes a target, it ceases to be a good measure"

Help support the show AND learn pytest: 

• The Complete pytest course is now a bundle, with each part available separately.
  • pytest Primary Power teaches the super powers of pytest that you need to learn to use pytest effectively.
  • Using pytest with Projects has lots of "when you need it" sections like debugging failed tests, mocking, testing strategy, and CI
  • Then pytest Booster Rockets can help with advanced parametrization and building plugins.
• Whether you need to get started with pytest today, or want to power up your pytest skills, PythonTest has a course for you.