Tests and verification with AI

Martin emphasizes rigorous testing, verifying LLM output, and warns about hallucinations and unreliable claims.

Play episode from 43:23

chevron_right

Transcript

chevron_right

Transcript

Episode notes

Brought to You By:

•⁠ Statsig ⁠ — ⁠ The unified platform for flags, analytics, experiments, and more. AI-accelerated development isn’t just about shipping faster: it’s about measuring whether, what you ship, actually delivers value. This is where modern experimentation with Statsig comes in. Check it out.

•⁠ Linear ⁠ — ⁠ The system for modern product development. I had a jaw-dropping experience when I dropped in for the weekly “Quality Wednesdays” meeting at Linear. Every week, every dev fixes at least one quality isse, large or small. Even if it’s one pixel misalignment, like this one. I’ve yet to see a team obsess this much about quality. Read more about how Linear does Quality Wednesdays – it’s fascinating!

—

Martin Fowler is one of the most influential people within software architecture, and the broader tech industry. He is the Chief Scientist at Thoughtworks and the author of Refactoring and Patterns of Enterprise Application Architecture, and several other books. He has spent decades shaping how engineers think about design, architecture, and process, and regularly publishes on his blog, MartinFowler.com.

In this episode, we discuss how AI is changing software development: the shift from deterministic to non-deterministic coding; where generative models help with legacy code; and the narrow but useful cases for vibe coding. Martin explains why LLM output must be tested rigorously, why refactoring is more important than ever, and how combining AI tools with deterministic techniques may be what engineering teams need.

We also revisit the origins of the Agile Manifesto and talk about why, despite rapid changes in tooling and workflows, the skills that make a great engineer remain largely unchanged.

—

Timestamps

(00:00) Intro

(01:50) How Martin got into software engineering

(07:48) Joining Thoughtworks

(10:07) The Thoughtworks Technology Radar

(16:45) From Assembly to high-level languages

(25:08) Non-determinism

(33:38) Vibe coding

(39:22) StackOverflow vs. coding with AI

(43:25) Importance of testing with LLMs

(50:45) LLMs for enterprise software