Adventures in DevOps

Share Episode ⸺ Episode Sponsor: Rootly AI - https://dev0ps.fyi/rootlyai                  Paul Conroy, CTO at Square1, joins the show to prove that the best defense against malicious bots isn't always a firewall—sometimes, it’s creative data poisoning. Paul recounts a legendary story from the Irish property market where a well-funded competitor attempted to solve their "chicken and egg" problem by scraping his company's listings. Instead of waiting years for lawyers, Paul’s team fed the scrapers "Project Yellow Brick Road": fake listings that placed the British Prime Minister at 10 Downing Street in Dublin and the White House in County Cork. The result? The competitor’s site went viral for all the wrong reasons, forcing them to burn resources manually filtering junk until they eventually gave up and targeted someone else.         >We also dive into the high-stakes world of election coverage, where Paul had three weeks to build a "coalition builder" tool for a national election. The solution wasn't a complex microservice architecture, but a humble Google Sheet wrapped in a Cloudflare Worker. Paul explains how they mitigated Google's rate limits and cold start times by putting a heavy cache in front of the sheet, leading to a crucial lesson in pragmatism: data that is "one minute stale" is perfectly acceptable if it saves the engineering team from building a complex invalidation strategy. Practically wins.         >Finally, the conversation turns to the one thing that causes more sleepless nights than malicious scrapers: caching layers. Paul and the host commiserate over the "turtles all the way down" nature of modern caching, where a single misconfiguration can lead to a news site accidentally attaching a marathon runner’s photo to a crime story. They wrap up with picks, including a history of cryptography that features the Pope breaking Spanish codes and a defense of North Face hiking boots that might just be "glamping" gear in disguise.         >🎯 Picks:         Warren - The North Face Hedgehog Gore-tex Hiking ShoesPaul - The Code Book

Share Episode                  "Those memes are not going to make themselves."         >Dorota, CEO of Authress, joins us to roast the 2025 DORA Report, which she argues has replaced hard data with an AI-generated narrative. From the confusing disconnect between feeling productive and actually shipping code to the grim reality of a 30% acceptance rate, Warren and Dorota break down why this year's report smells a lot like manure.         >We dissect the massive 142-page 2025 DORA Report. Dorota argues that the report, which is now rebranded as the "State of AI-Assisted Software Development", feels less like a scientific study of DevOps performance and more like a narrative written by an intern using an LLM prompt. The duo investigates the "stubborn results" where AI apparently makes everyone feel like a 10x developer, where the hard results tell a different story. AI actually increases software and product instability — failing to improve.         >The conversation gets spicy as they debate the "pit of failure" that is feature flags (often used as a crutch for untested code) and the embarrassing reality that GitHub celebrates a mere 30% code acceptance rate as a "success." Dorota suggests that while AI raises the floor for average work, it completely fails when you need to solve complex problems or, you know, actually collaborate with another human being.         >In a vivid analogy, Dorota compares reading this year's report to the Swiss Spring phenomenon — the time of year when farmers spray manure, leaving the beautiful landscape smelling...unique. The episode wraps up with a reality check on the physical limits of LLM context windows (more tokens, more problems) and a strong recommendation to ignore the AI hype cycle in favor of a much faster-growing organism: a kitchen countertop oyster mushroom kit.         >💡 Notable Links:         AI as an amplifier truism fallacyDORA 2025 ReportDevOps Episode: VS Code & GitHub CopilotWhere is the deluge of new software - Impact of AI on software productsImpact of AI on Critical Thinking🎯 Picks:         Warren - The Maximum Effective Context WindowDorota - Mushroom Grow Kit

Share Episode ⸺ Episode Sponsor: Incident.io - https://dev0ps.fyi/incidentio                  "My biggest legacy at Google is the amount of systems I broke." — Sam Goto joins the show with a name that strikes fear into engineering systems everywhere. As a Senior Staff Engineer on the Chrome team, Sam shares the hilarious reality of having the last name "Goto," which once took down Google's internal URL shortener for four hours simply because he plugged in a new computer.         >Sam gets us up to speed with Federated Credentials Management (FedCM), as we dive deep into why authentication has been built despite the browser rather than with it, and why it’s time to move identity from "user-land" to "kernel-land". This shift allows for critical UX improvements for logging in all users irrespective of what login providers you use, finally addressing the "NASCAR flag" problem of infinite login lists.         >Most importantly, he shares why you don't need to change your technology stack to get all the benefits of FedCM. Finally, Sam details the "self-sustaining flame" strategy (as opposed to an ecosystem "flamethrower"), revealing how they utilized JavaScript SDKs to migrate massive platforms like Shopify and 50% of the web's login traffic without requiring application developers to rewrite their code.         >💡 Notable Links:         HSMs + TPM in production environmentsGet involved: FedCM W3C WGThe FedCM spec GitHub repoTPAC Browser Conference🎯 Picks:         Warren - Book: The Platform RevolutionSam - The 7 Laws of Identity and Short Story: The Egg By Andy Weir

Elise Stanley Breval, VP and Head of UX at Unleash, brings 30 years of experience in user-centered design to the conversation. She challenges the misconception that UX is merely about aesthetics, discussing the critical friction between engineering and customer needs. Elise shares a memorable story of overcoming misguided branding decisions with practical user research. They also debate the role of engineers in user interactions and highlight the importance of fostering a culture that values feedback and collaboration in product development.

Share Episode ⸺ Episode Sponsor: Incident.io - https://dev0ps.fyi/incidentioWarren is joined by Olga Kundzich, Co-founder and CTO of Moderne, to discuss the reality of technical debt in modern software engineering. Olga reveals a shocking statistic: without maintenance, cloud-native applications often cease to function within just six months. And from our experience, that's actually optimistic. The rapid decay isn't always due to bad code choices, but rather the shifting sands of third-party dependencies, which make up 80 to 90% of cloud-native environments.We review the limitations of traditional Abstract Syntax Trees (ASTs) and the introduction of OpenRewrite's Lossless Semantic Trees (LSTs). Unlike standard tools, LSTs preserve formatting and style, allowing for automated, horizontal scaling of code maintenance across millions of lines of code. This fits perfectly in to the toolchain that is the LLMs and open source ecosystem. Olga explains how this technology enables enterprises to migrate frameworks—like moving from Spring Boot 1 to 2 — without dedicating entire years to manual updates.Finally, they explore the intersection of AI and code maintenance, noting that while LLMs are great at generating code, they often struggle with refactoring and optimizing existing codebases. We highlight that agents are not yet fully autonomous and will always require "right-sized" data to function effectively. Will is absent for this episode, leaving Warren to navigate the complexities of mass-scale code remediation solo.💡 Notable Links:DevOps Episode: We read codeDevOps Episode: Dynamic PRs from incidentsOpenRewriteLarger Context Windows are not better🎯 Picks:Warren - Dell XPS 13 9380Olga - Claude Code

John Papa, Partner General Manager at Microsoft and former Disney architect, shares his journey in developer tools and the creation of VS Code. He discusses 'Agentic AI,' predicting a shift where developers become managers of AI agents, and warns about the risks of automation. John recounts cautionary tales from Disney, emphasizing the perils of skipping testing and the importance of human oversight. He challenges listeners to question the necessity of new AI features, leaving them with the thought: 'Who asked for that?'

Share Episode ⸺ Episode Sponsor: Attribute - https://dev0ps.fyi/attributeIn the wake of one of the worst AWS incidents in history, we're joined by Lawrence Jones, Founding Engineer at Incident.io. The conversation focuses on the challenges of managing incidents in highly regulated environments like FinTech, where the penalties for downtime are harsh and require a high level of rigor and discipline in the response process. Lawrence details the company's evolution, from running a monolithic Go binary on Heroku to moving to a more secure, robust setup in GCP, prioritizing the use of native security primitives like GCP Secret Manager and Kubernetes to meet the obligations of their growing customer base.We spotlight exactly how a system can crawl GitHub pull requests, Slack channels, telemetry data, and past incident post-mortems to dynamically generate an ephemeral runbook for the current incident.Also discussed are the technical challenges of using RAG (Retrieval-Augmented Generation), noting that they rely heavily on pre-processing data with tags and a service catalog rather than relying solely on less consistent vector embeddings to ensure fast, accurate search results during a crisis.Finally, Lawrence stresses that frontier models are no longer the limiting factor in building these complex systems; rather, success hinges on building structured, modular systems, and doing the hard work of defining objective metrics for improvement.💡 Notable Links:Cloud Secrets management at scaleEpisode: Solving Time Travel in RAG DatabasesEpisode: Does RAG Replace keyword search?🎯 Picks:Warren - Anker Adpatable Wall-Charger - PowerPort Atom IIILawrence - Rocktopus & The Checklist Manifesto

Share Episode ⸺ Episode Sponsor: Attribute - https://dev0ps.fyi/attributeWe're joined by 20 year industry veteran and DevOps advocate, Adam Korga, celebrating the release of his book IT Dictionary. In this episode we quickly get down to the inspiration behind postmortems as we review some cornerstone cases both in software and in general technology.Adam shares how he started in the industry, long before DevOps was a coined term, focused on making systems safer and avoiding mistakes like accidentally dropping a production database. we review the infamous incidents of accidental database deletion, by LLMs and human's alike.And of course we touch on the quintessential postmortems in civil engineering, flight, and survivorship bias from World War II through analyzing bullet holes on returning planes.💡 Notable Links:Adam's book: IT DictionaryKnight Capital: the 45 minute nightmareWork Chronicles Comic: Will my architecture work for 1 Million users?🎯 Picks:Warren - Cuitisan CANDL storage containersAdam - FUBAR

Share Episode ⸺ Episode Sponsor: Attribute - https://dev0ps.fyi/attributeJenna Pederson, Staff Developer Relations at Pinecone, joins us to close the loop on Vector Databases. Demystifies how they power semantic search, their role in RAG, and also unexpected applications.Jenna takes us beyond the buzzword bingo, explaining how vector databases are the secret sauce behind semantic search. Sharing just how "red shirt" gets converted into a query that returns things semantically similar. It's all about turning your data into high-dimensional numerical meaning, which, as Jenna clarifies, is powered by some seriously clever math to find those "closest neighbors."The conversation inevitably veers into Retrieval-Augmented Generation (RAG). Jenna reveals how databases are the unsung heroes giving LLMs real brains (and up-to-date info) when they're prone to hallucinating or just don't know your company's secrets. They complete the connection from proprietary and generalist foundational models to business relevant answers.💡 Notable Links:Episode: MCP: The Model Context Protocol and Agent InteractionsCrossing the Chasm🎯 Picks:Warren - HanCenDa USB C Magnetic adapterJenna - Keychron Alice Layout Mechanical keyboard (And get a 5% discount on us)

Share EpisodeThis episode we are joined by Andrew Moreland, co-founder of Chalk. Andrew explains how their company's core business model is to deploy their software directly into their customers' cloud environments. This decision was driven by the need to handle highly sensitive data, like PII and financial records, that customers don't want to hand over to a third-party startup. The conversation delves into the surprising and complex challenges of this approach, which include managing granular IAM permissions and dealing with hidden global policies that can block their application. Andrew and Warren also discuss the real-world network congestion issues that affect cross-cloud traffic, a problem they've encountered multiple times. Andrew shares Chalk's mature philosophy on software releases, where they prioritize backwards compatibility to prevent customer churn, which is a key learning from a competitor.Finally, the episode explores the advanced technical solutions Chalk has built, such as their unique approach to "bitemporal modeling" to prevent training bias in machine learning datasets. As well as, the decision to move from Python to C++ and Rust for performance, using a symbolic interpreter to execute customer code written in Python without a Python runtime. The episode concludes with picks, including a surprisingly popular hobby and a unique take on high-quality chocolate.💡 Notable Links:Fact - The $1M hidden Kubernetes spendGiraffe and Medical Ruler training data biasSOLID principles don't produce better code?Veritasium - The Hole at the Bottom of MathEpisode: Auth Showdown on backwards compatible changes🎯 Picks:Warren - Switzerland Grocery Store ChocolateAndrew - Trek E-Bikes