The Backend Engineering Show with Hussein Nasser

In this video, I go through how I overcome procrastination as a software engineer.  What is Procrastination? 0:00How to Defeat  * Reward based system - a reward after achieving  1:20* discipline, remembering why started this, your goal 3:16* Professional - I need to do the work and ship 6:10

2020 retrospective Intro 0:00 Goals 0:50 Teaching vs Documenting 4:30 Channel Growth 7:40 2021 10:50

0:00 Intro 1:00 Summary of the Outage 4:00 Detailed Analysis of the Incident Report On Dec 14 2020 Google across the globe suffered from an outage that lasted 45 minutes nobody could access most of Google services. Google has released a detailed incident report discussing the outage, what caused it, technical details on their internal service architecture and what did they do to mitigate and prevent this from happening in this in the future In this video, I want to take a few minutes to summarize the report and then go into a detailed analysis. You can find youtube chapters to jump to the interesting part of the video. pick your favorite drink, sit back relax, and enjoy. Let's get started. let's start with an overview of how the google id service works, the client connects to Google authentication service to get authenticated or retrieve account information The account information is stored in a distributed manner between the different service ids for redundancy. when an update is made to an account on the leader node, the existing data in all nodes are marked as outdated, this is done for security reasons. Let’s say you updated your credit card info, privated your profile or deleted a comment, it is extremely dangerous to serve that outdated information. This was the key to the outage. The updated account is then replicated based on Paxos Consensus protocol. The user id service has a storage quota controlled by an automated quota management solution when the storage usage of the service changes. the quota is maintained accordingly either reduced or increased based on the demand .. So What Exactly Happened that caused the outage? In October 2020, google migrated their quota management to a new system and registered the id service with the new system. however some parts of the old system remained hooked up specifically the parts regarding the reading of the service usage. And because the service is registered to the new system, the old qouta system reported 0 usage as it should. So when the new quota manement asked its service for its usage it was incorrectly reporting 0. Nothing happened for a while since there was a grace period, but that period expired on December Thats when the new quota system kicked and saw the id service with 0 usage and started reducing the qouta for the id service down .. you are not using it why waste? The quota kept reducing until the service had no space left. This has caused updates to the leader node to fail, which caused all data to go out of date in all nodes which in turn escalated globally to what we have seen. Resource https://status.cloud.google.com/incident/zall/20013

The Backend Engineering Show Live with Hussein Nasser episode 10 we discuss many great questions!! Indexing Woes, The Secret to Backend Interviews, What is on my Bookshelf? Backend Engineering Show

Exposed Postgres instances are being ssh into and used as a botnet to mine bitcoin, in this video we explain how does that happens.   the trick is the COPY FROM PROGRAM command

At 3:47 am PST almost all google services went down including, gmail, youtube, drive, docs, meet, nest , google maps and many more. It took close to an hour to bring them back up. We still don’t know what caused this outage, in this video we will try to make sense from what we have gathered so far. A detailed analysis video will follow once we get a response from google symptoms Could not sign in to google (account not found) Could not authenticate if you already have a token Services not require authentication also fails to retrieve certain account information (profile, YouTube comments) Guess -> Borg Service that provide authentication, authorization went down, a fix? Storage qouta issue? https://twitter.com/googlecloud/status/1338493015145504770 https://www.tomsguide.com/news/gmail-and-youtube-down-several-google-services-are-not-working-latest-updates Google down? Realtime status, issues and outages for the UK | Downdetector Google Workspace Status Dashboard

When the private key of a matching public key that belong to a certificate is leaked, an attacker can intercept server hello, use their own dh parameters sign it with the stolen private key and ship it to the client effectively doing MITM. This is extremely dangerous and we have no way in the client to know a MITM has happened. That is why a certificate sometimes has to be revoked, and in this video I’m going to discuss those revocation techniques. 0:00 How Certificate Works 3:00 Certificate Revocation List 4:10 OCSP 7:00 OCSP Stapling

In The Backend Engineering Show Live, we discuss Impostor syndrome and Staying Motivated in software engineering field.

Oblivious DoH is a technology that separates IP addresses from queries, so that no single entity can see both at the same time. Cloudflare, Apple & Fastly worked on this and did a good write-up of the tech, we discuss it in this video https://blog.cloudflare.com/oblivious-dns/ https://blog.cloudflare.com/oblivious-dns/

Oracle introduces a Game Changer Feature in MySQL that allows for OLAP & OLTP workloads in a single database. This is huge let us discuss https://www.oracle.com/emea/news/announcement/oracle-announces-mysql-database-service-with-integrated-analytics-engine-2020-12-03.html https://dev.mysql.com/doc/mysql-analytics/en/mysql-analytics-introduction.html 0:00 Intro 1:40 History of ETL 7:00 How Kafka Helped Data Warehouse 8:20 How RAPID Solves this 11:14 MySQL Database Service Analytics Engine (RAPID) Architecture 14:00 Loading Data 18:00 Summary