The Changelog: Software Development, Open Source

Kaizen! The day half the internet went down (Interview)

Aug 5, 2021

The hosts dive into the Kaizen philosophy, emphasizing continuous improvement in tech and podcasting. They reflect on the significant global Fastly outage and its impact, advocating for multi-cloud strategies and better redundancy. The discussion also covers incident management challenges, focusing on access tokens and the importance of documentation. Expect insights on innovating through experimentation in development, alongside strategies for improving usability and community engagement in software projects. It's a deep yet practical look at enhancing performance.

Ask episode

AI Snips

Chapters

Transcript

Episode notes

ADVICE

Kaizen for Self-Improvement

Embrace Kaizen, the art of continuous self-improvement.
Regularly reflect on how to improve processes and products.

ANECDOTE

Fastly Outage Impact

Fastly's outage took down major websites like BBC and impacted Slack's emoji functionality.
Changelog.com was also down for 21 minutes due to this outage.

INSIGHT

Redundancy Considerations

Redundancy is crucial but costly; Changelog.com lost listener stats for a popular episode due to the Fastly outage.
Evaluate redundancy costs versus potential gains.

Get the Snipd Podcast app to discover more snips from this episode

Get the app

This week we’re sharing a special episode of our new podcast called Ship It. This episode is our Kaizen-style episode where we point our lens inward to Changelog.com to see what we should improve next. The plan is do this episode style every 10 episodes.

Gerhard, Adam, and Jerod talk about the things that we want to improve in our setup over the next few months. We talk about how the June Fastly outage affected changelog.com, how we responded that day, and what we could do better. We discuss multi-cloud, multi-CDN, and the next sensible and obvious improvements for our app.

Join the discussion

Changelog++ members get a bonus 1 minute at the end of this episode and zero ads. Join today!

Sponsors:

InfluxDB – InfluxDB empowers developers to build IoT, analytics, and monitoring software. It’s purpose-built to handle massive volumes and countless sources of time-stamped data produced by sensors, applications, and infrastructure. Learn about the wide range of use cases of InfluxDB at influxdata.com/changelog
LaunchDarkly – Ship fast. Rest easy. Deploy code at any time, even if a feature isn’t ready to be released to your users. Wrap code in feature flags to get the safety to test new features and infrastructure in prod without impacting the wrong end users.
Sentry – Working code means happy customers. That’s exactly why teams choose Sentry. From error tracking to performance monitoring, Sentry helps teams see what actually matters, resolve problems quicker, and learn continuously about their applications - from the frontend to the backend. Use the code SHIPIT and get the team plan free for three months.
Grafana Cloud – Our dashboard of choice Grafana is the open and composable observability and data visualization platform. Visualize metrics, logs, and traces from multiple sources like Prometheus, Loki, Elasticsearch, InfluxDB, Postgres and many more.

Featuring:

Gerhard Lazu – Website, GitHub, LinkedIn, X
Jerod Santo – GitHub, LinkedIn, Mastodon, X
Adam Stacoviak – Website, GitHub, LinkedIn, Mastodon, X

Show Notes:

What does Kaizen mean? (Wikipedia)
Fastly 8th of June outage took offline The Guardian, CNN, New York Times, BBC and Changelog. Slack, Amazon, Stackoverflow and many others were also affected
This is how it all started
@danhett becomes famous on Twitter because of this tweet
WIRED: What really went down when the internet went down
BLOOMBERG: What Is Fastly and Why a Slew of Websites Went Offline

Something missing or broken? PRs welcome!