Ship It! Cloud, SRE, Platform Engineering

This week Gerhard is joined by Justin Searls, Test Double co-founder and CTO. Also a 🐞 magnet. They talk about how to deal with the pressure of shipping faster, why you should optimize for smoothness not speed, and why focusing on consistency is key. Understanding the real why behind what you do is also important. There’s a lot more to it, as its a nuanced and complex discussion, and well worth your time. Expect a decade of learnings compressed into one hour, as well as disagreements on some ops and infrastructure topics — all good fun. In the show notes, you will find Gerhard’s favorite conference talks Justin gave a few years back. Join the discussionChangelog++ members get a bonus 1 minute at the end of this episode and zero ads. Join today!Sponsors:Fly – Deploy your apps and databases close to your users. In minutes you can run your Ruby, Go, Node, Deno, Python, or Elixir app (and databases!) all over the world. No ops required. Learn more at fly.io/changelog and check out the speedrun in their docs. LaunchDarkly – Ship fast. Rest easy. Deploy code at any time, even if a feature isn’t ready to be released to your users. Wrap code in feature flags to get the safety to test new features and infrastructure in prod without impacting the wrong end users. SignalWire – Build what’s next in communications with video, voice, and messaging APIs powered by elastic cloud infrastructure. Try it today at signalwire.com and use code SHIPIT for $25 in developer credit. Grafana Cloud – Our dashboard of choice Grafana is the open and composable observability and data visualization platform. Visualize metrics, logs, and traces from multiple sources like Prometheus, Loki, Elasticsearch, InfluxDB, Postgres and many more. Featuring:Justin Searls – Website, GitHub, Mastodon, XGerhard Lazu – Website, GitHub, LinkedIn, XShow Notes:⭐️ Justin is giving the keynote at this year’s Reliable Web Summit. The topic is why distrust is at the heart of a lot of the issues we discussed and how to build trust as an individual, small team, and organization. ⭐️ It all started with this Tweet 5 for 5000: Find your Leading Indicators 🎬 The Selfish Programmer 🎬 Running a business, demystified 🎬 How to program The most impressive Olympic event I’ve seen out of Tokyo so far Justin Searls: If you’re a programmer and you’re happy with the work you’re doing, you’re growing in the ways you want, and you feel pushed to do your best work as opposed to work the most hours, congratulations! That’s all too rare. Something missing or broken? PRs welcome!

In this episode, Gerhard follows up on The Changelog #375, which is the last time that he spoke Crossplane with Dan and Jared. Many things changed since then, such as abstractions and compositions, as well as using Crossplane to build platforms, which were mostly ideas. Fast forward 18 months, 2k changes, as well as a major version, and Crossplane is now an easy choice - some would say the best choice - for platform teams to declare what infrastructure means to them. You can now use Crossplane to define your infrastructure abstractions across multiple vendors, including AWS, GCP & Equinix Metal. The crazy ideas from 2019 are now bold and within reach. Gerhard also has an idea for the changelog.com 2022 setup. Listen to what Jared & Dan think, and then let us know your thoughts too. Join the discussionChangelog++ members get a bonus 1 minute at the end of this episode and zero ads. Join today!Sponsors:Fly – Deploy your apps and databases close to your users. In minutes you can run your Ruby, Go, Node, Deno, Python, or Elixir app (and databases!) all over the world. No ops required. Learn more at fly.io/changelog and check out the speedrun in their docs. Sentry – Working code means happy customers. That’s exactly why teams choose Sentry. From error tracking to performance monitoring, Sentry helps teams see what actually matters, resolve problems quicker, and learn continuously about their applications - from the frontend to the backend. Use the code SHIPIT and get the team plan free for three months. SignalWire – Build what’s next in communications with video, voice, and messaging APIs powered by elastic cloud infrastructure. Try it today at signalwire.com and use code SHIPIT for $25 in developer credit. Grafana Cloud – Our dashboard of choice Grafana is the open and composable observability and data visualization platform. Visualize metrics, logs, and traces from multiple sources like Prometheus, Loki, Elasticsearch, InfluxDB, Postgres and many more. Featuring:Dan Mangum – Website, GitHub, XJared Watts – GitHub, XGerhard Lazu – Website, GitHub, LinkedIn, XShow Notes: Changelog #375 Why Crossplane is so Exciting Crossplane vs Terraform 🎬 Day 2 Operations on Crossplane - Dan Mangum, July 2021 🎬 How to Build Your Own PaaS from the Cloud Native Landscape - Jared Watts, May 2021 🎬 Crossplane Community Day Keynote - Kelsey Hightower, May 2020 Something missing or broken? PRs welcome!

In today’s episode, Gerhard is joined by Uma, CEO and co-founder of ChaosNative, as well as Karthik, CTO and also a ChaosNative co-founder. They talk Chaos Engineering and Litmus. Chaos Engineering is not just for super SREs. It is not meant to prevent outages. And, it is not just about hardware. Chaos Engineering is about testing how reliable your systems are. It’s meant to show you how things fail, including when other dependent systems fail - think cascading failures. This is a good way to discover inconvenient truths about that beautiful code that you wrote. Everything fails, and great insights are to be found when it does. Join the discussionChangelog++ members get a bonus 1 minute at the end of this episode and zero ads. Join today!Sponsors:Render – The Zero DevOps cloud that empowers you to ship faster than your competitors. Render is built for modern applications and offers everything you need out-of-the-box. Learn more at render.com/changelog or email changelog@render.com for a personal introduction and to ask questions about the Render platform. Sentry – Working code means happy customers. That’s exactly why teams choose Sentry. From error tracking to performance monitoring, Sentry helps teams see what actually matters, resolve problems quicker, and learn continuously about their applications - from the frontend to the backend. Use the code SHIPIT and get the team plan free for three months. Teleport – Teleport Access Plane lets you access any computing resource anywhere. Engineers and security teams can unify access to SSH servers, Kubernetes clusters, web applications, and databases across all environments. Try Teleport today in the cloud, self-hosted, or open source at goteleport.com Fastly – Our bandwidth partner. Fastly powers fast, secure, and scalable digital experiences. Move beyond your content delivery network to their powerful edge cloud platform. Learn more at fastly.com. Featuring:Uma Mukkara – Website, GitHub, LinkedIn, XKarthik Satchitanand – GitHub, LinkedIn, XGerhard Lazu – Website, GitHub, LinkedIn, XShow Notes: Defining Chaos Engineering & Its Principles - 7 mins read Principles of Cloud Native Chaos Engineering - 10 mins read Litmus - Chaos Engineering for your Kubernetes 🎬 SLOconf: Benchmarking SLOs using Chaos Engineering - Uma Mukkara 🎬 Putting Chaos into CD to Increase App Resiliency - Juergen Etzlstorfer & Karthik Satchitanand Chaos Carnival 2021 🎬 ChaosCarnival Day 1 Keynote - Adrian Cockcroft Something missing or broken? PRs welcome!

In this episode, Gerhard talks to his Skyhook Adventure friends: Alan Cooney, Saul Cullen & Wycliffe Maina. They are the ones that introduced Gerhard to the world of serverless in the context of Amazon Web Services. Gerhard shared his experience with remote work, how to ship software with confidence and consistency, and what to look for in infrastructure as code. At the heart of Skyhook Adventure are adventure trips, and 2020 was not a good one for this business. As you can already tell, code and infrastructure was not the biggest challenge for this team. Having said that, serverless, microservices, a monorepo and the event-based architecture played a big part in successfully navigating the challenges. This is a story about what happens when a good team allows itself to be guided by solid experience and keeps doing the right thing, long-term. It’s fun, real, and it applies to many. Join the discussionChangelog++ members get a bonus 1 minute at the end of this episode and zero ads. Join today!Sponsors:Render – The Zero DevOps cloud that empowers you to ship faster than your competitors. Render is built for modern applications and offers everything you need out-of-the-box. Learn more at render.com/changelog or email changelog@render.com for a personal introduction and to ask questions about the Render platform. LaunchDarkly – Ship fast. Rest easy. Deploy code at any time, even if a feature isn’t ready to be released to your users. Wrap code in feature flags to get the safety to test new features and infrastructure in prod without impacting the wrong end users. Sentry – Working code means happy customers. That’s exactly why teams choose Sentry. From error tracking to performance monitoring, Sentry helps teams see what actually matters, resolve problems quicker, and learn continuously about their applications - from the frontend to the backend. Use the code SHIPIT and get the team plan free for three months. Grafana Cloud – Our dashboard of choice Grafana is the open and composable observability and data visualization platform. Visualize metrics, logs, and traces from multiple sources like Prometheus, Loki, Elasticsearch, InfluxDB, Postgres and many more. Featuring:Alan Cooney – Website, GitHub, LinkedInSaul Cullen – GitHub, LinkedIn, XWycliffe Maina – Website, GitHub, LinkedIn, XGerhard Lazu – Website, GitHub, LinkedIn, XShow Notes: Skyhook Adventure - the end-result that this team ships many times per day 📚Welcome to the Experience Economy (the book that @saulgcullen mentioned) The skyhookadventure.com 2021 setup: AWS Lambda - serverless functions AWS Event Bridge - serverless events AWS DynamoDB - serverless document based database AWS CDK - infrastructure as code Vercel - server-side rendered frontend hosting GitHub - version control GitHub Actions - continuous delivery Split - feature flags Drip - email marketing Something missing or broken? PRs welcome!

Gerhard talks to Tom Wilkie, VP of Product for Grafana Labs. They talk about Loki, Tempo, and how can Grafana Cloud offer such a generous free tier. The solution is in the Cortex architecture, which was used in Loki and in Tempo too. Yes, Tom is the Cortex co-author. We recommend that you listen to this episode in combination with episodes 3 and 11. That’s the best way to get a more complete picture of the topics that we discuss today. Lastly, would you like to watch Gerhard & Tom pair-up and build Grafana dashboards like pros? Tom has this really interesting approach that Gerhard would like to learn too. We can either have a live YouTube stream, or record and then publish the video. Let us know your preference via our Changelog Slack, or just plain Twitter. Join the discussionChangelog++ members get a bonus 1 minute at the end of this episode and zero ads. Join today!Sponsors:Fly – Deploy your apps and databases close to your users. In minutes you can run your Ruby, Go, Node, Deno, Python, or Elixir app (and databases!) all over the world. No ops required. Learn more at fly.io/changelog and check out the speedrun in their docs. LaunchDarkly – Ship fast. Rest easy. Deploy code at any time, even if a feature isn’t ready to be released to your users. Wrap code in feature flags to get the safety to test new features and infrastructure in prod without impacting the wrong end users. Cockroach Labs – Scale fast, survive anything, thrive everywhere! CockroachDB is most highly evolved database on the planet. Build and scale fast with CockroachCloud (CockroachDB hosted as a service) where a team of world-class SREs maintains and manages your database infrastructure, so you can focus less on ops and more on code. Get started for free their 30-day trial or try their forever-free tier. Learn more at cockroachlabs.com/changelog. Fastly – Our bandwidth partner. Fastly powers fast, secure, and scalable digital experiences. Move beyond your content delivery network to their powerful edge cloud platform. Learn more at fastly.com. Featuring:Tom Wilkie – GitHub, XGerhard Lazu – Website, GitHub, LinkedIn, XShow Notes: Cortex Loki Tempo grafana-agent Grafana Cloud Grafana & Prometheus unified alerting What Does the Future Hold for Observability? Kubecon EU 2019 Keynote Something missing or broken? PRs welcome!

Gerhard talks with Charity Majors, ops engineer and accidental startup founder at honeycomb.io about high-performing teams, why “15 minutes or bust,” and how we should start using Honeycomb in our own monolithic Phoenix app that runs changelog.com. There is just one step, and it’s actually really simple! They also talk about how Honeycomb uses Honeycomb to learn about Honeycomb, which is one of Gerhard’s favorite questions. As for key take-aways, deploying straight into production is really important, but not as important as optimising for humans - which are not replaceable cogs, that learn and share their learnings continuously. That is the secret to making things easy and happy for everyone. Join the discussionChangelog++ members save 5 minutes on this episode because they made the ads disappear. Join today!Sponsors:Fly – Deploy your apps and databases close to your users. In minutes you can run your Ruby, Go, Node, Deno, Python, or Elixir app (and databases!) all over the world. No ops required. Learn more at fly.io/changelog and check out the speedrun in their docs. Armory – Empower your development teams to deploy code with increased safety, resilience, velocity, and compliance – to any production target on prem or in the cloud using Armory’s enterprise-grade distribution of Spinnaker. Learn more at armory.io/shipit LaunchDarkly – Ship fast. Rest easy. Deploy code at any time, even if a feature isn’t ready to be released to your users. Wrap code in feature flags to get the safety to test new features and infrastructure in prod without impacting the wrong end users. Fastly – Our bandwidth partner. Fastly powers fast, secure, and scalable digital experiences. Move beyond your content delivery network to their powerful edge cloud platform. Learn more at fastly.com. Featuring:Charity Majors – Website, GitHub, XGerhard Lazu – Website, GitHub, LinkedIn, XShow Notes: 📚 Observability Engineering (Early Release) - Free download 🎬 The State of Observability 2021: Mature Teams Ship Better Code Faster and You Can Too What is Observability? Define SLOs Kubernetes is complicated rubygems.org Fastly logs in Honeycomb RDS Performance Degradation – Postmortem 2018 o11ycon 2021 videos hnycon 2021 keynote Something missing or broken? PRs welcome!

Kaizen means “change for the better”, continuous improvement in this context. Failure is essential to learning, but how do we learn as a team? The simplest thing is to regularly dedicate time for taking a step back, talking about what works & what doesn’t, maybe writing some of it down, and eventually deciding what we should improve next. I intend to make every 10th Ship It! episode a Kaizen one. This is the first one when we talk with Adam and Jerod about the things that we want to improve in our setup over the next few months. We talk about how the June Fastly outage affected changelog.com, how we responded that day, and what we could do better. We discuss multi-cloud, multi-CDN, and the next sensible and obvious improvements for our app. Let us know via Slack or Twitter what learnings are valuable to you so that we can produce the best content for you. Join the discussionChangelog++ members get a bonus 2 minutes at the end of this episode and zero ads. Join today!Sponsors:Fly – Deploy your apps and databases close to your users. In minutes you can run your Ruby, Go, Node, Deno, Python, or Elixir app (and databases!) all over the world. No ops required. Learn more at fly.io/changelog and check out the speedrun in their docs. Teleport – Teleport Access Plane lets you access any computing resource anywhere. Engineers and security teams can unify access to SSH servers, Kubernetes clusters, web applications, and databases across all environments. Try Teleport today in the cloud, self-hosted, or open source at goteleport.com Linode – Get $100 in free credit to get started on Linode – Linode is our cloud of choice and the home of Changelog.com. Head to linode.com/changelog OR text CHANGELOG to 474747 to get instant access to that $100 in free credit. Fastly – Our bandwidth partner. Fastly powers fast, secure, and scalable digital experiences. Move beyond your content delivery network to their powerful edge cloud platform. Learn more at fastly.com. Featuring:Adam Stacoviak – Website, GitHub, LinkedIn, Mastodon, XJerod Santo – GitHub, LinkedIn, Mastodon, XGerhard Lazu – Website, GitHub, LinkedIn, XShow Notes: What does Kaizen mean? (Wikipedia) Fastly 8th of June outage took offline The Guardian, CNN, New York Times, BBC and Changelog. Slack, Amazon, Stackoverflow and many others were also affected This is how it all started @danhett becomes famous on Twitter because of this tweet WIRED: What really went down when the internet went down BLOOMBERG: What Is Fastly and Why a Slew of Websites Went Offline Something missing or broken? PRs welcome!

This week we talk with Jean-Sébastien Pedron, RabbitMQ and FreeBSD contributor, about the importance of good release engineering for core infrastructure. Both Jean-Sébastien and I have been part of the Core RabbitMQ team for many years now. We have built some of the biggest CI/CD pipelines (check the show notes for one example), wrote and shipped some great code together, while breaking and fixing many things in the process. We have been wrestling with today’s topic since 2016. Jean-Sébastien has some great FreeBSD stories to share, as well as an interesting perspective on shipping graphic card drivers. Oh, and by the way, it’s probably our fault why your remote car key stopped working that afternoon. It will all make sense after you listen to this episode. Join the discussionChangelog++ members get a bonus 2 minutes at the end of this episode and zero ads. Join today!Sponsors:Fly – Deploy your apps and databases close to your users. In minutes you can run your Ruby, Go, Node, Deno, Python, or Elixir app (and databases!) all over the world. No ops required. Learn more at fly.io/changelog and check out the speedrun in their docs. Teleport – Teleport Access Plane lets you access any computing resource anywhere. Engineers and security teams can unify access to SSH servers, Kubernetes clusters, web applications, and databases across all environments. Try Teleport today in the cloud, self-hosted, or open source at goteleport.com Linode – Get $100 in free credit to get started on Linode – Linode is our cloud of choice and the home of Changelog.com. Head to linode.com/changelog OR text CHANGELOG to 474747 to get instant access to that $100 in free credit. LaunchDarkly – Ship fast. Rest easy. Deploy code at any time, even if a feature isn’t ready to be released to your users. Wrap code in feature flags to get the safety to test new features and infrastructure in prod without impacting the wrong end users. Featuring:Jean-Sébastien Pedron – GitHubGerhard Lazu – Website, GitHub, LinkedIn, XShow Notes: FreeBSD release engineering Netflix and FreeBSD: Using open source to deliver streaming video List of products based on FreeBSD Darktable Mesa Erlang hot code reloading: Appup cookbook Target system principles RabbitMQ v3.8 CI/CD pipeline in 2020 - each square is either a test suite, build or publish step Something missing or broken? PRs welcome!

Explore the essentials of Cloud Native, including Kubernetes, Docker, and CI/CD pipelines. Learn about transitioning to cloud services, microservices optimization with Kubernetes, and automated pipelines for continuous delivery. Discover the importance of monitoring in Cloud-Native development and community engagement in Cloud Native practice.

This week on Ship It! Gerhard talks with Lars Wikman (independent Elixir/BEAM software consultant) why sometimes a monolith running on a single host with continuous backups and a built-in self-restore capability is everything that a small team of developers needs. That’s right, no Kubernetes or microservices. After 2 years of running changelog.com, a Phoenix monolith, on Kubernetes, what do I think? Join our discuss and find out! Join the discussionChangelog++ members save 3 minutes on this episode because they made the ads disappear. Join today!Sponsors:Render – The Zero DevOps cloud that empowers you to ship faster than your competitors. Render is built for modern applications and offers everything you need out-of-the-box. Learn more at render.com/changelog or email changelog@render.com for a personal introduction and to ask questions about the Render platform. Teleport – Teleport Access Plane lets you access any computing resource anywhere. Engineers and security teams can unify access to SSH servers, Kubernetes clusters, web applications, and databases across all environments. Try Teleport today in the cloud, self-hosted, or open source at goteleport.com Cockroach Labs – Scale fast, survive anything, thrive everywhere! CockroachDB is most highly evolved database on the planet. Build and scale fast with CockroachCloud (CockroachDB hosted as a service) where a team of world-class SREs maintains and manages your database infrastructure, so you can focus less on ops and more on code. Get started for free their 30-day trial or try their forever-free tier. Learn more at cockroachlabs.com/changelog. Featuring:Lars Wikman – Website, GitHub, XGerhard Lazu – Website, GitHub, LinkedIn, XShow Notes: 🎬Follow-up YouTube stream Post-mortem: 10 years in the vertical: Part 1 Beam Bloggers Webring Litestream - Continuously stream SQLite changes to an object store Teaching Elixir & other videos Live stream - PromEx w. Alex Koutmos Regular Programming, conversations about software development and programming BEAM Radio - Stories and conversations around Elixir, Erlang and the BEAM ecosystem Something missing or broken? PRs welcome!