The Backend Engineering Show with Hussein Nasser

Pinterest moves to HTTP/3 on all their clients and edge CDNs this year. They witnessed interesting gains but not without good lesson learned. The main one was the mismatch of alt-svc vs DNS ttls. I cover this on the next episode of the backend engineering course. 0:00 Intro 2:00 Moving h2 to h3 through alt-svc 5:00 Why HTTP/3 6:00 HTTP/1 vs HTTP/2 9:00 TCP Head of Line blocking in HTTP/2 11:00 How HTTP/3 addresses HOL 12:15 Connection Migration 13:30 Stream level congestion control 14:10 1-RTT - 0-RTT 15:41 Pinterest challenges moving HTTP/3 19:00 Migration 21:15 Future work 22:30 Summary article https://medium.com/pinterest-engineering/pinterest-is-now-on-http-3-608fb5581094 Fundamentals of Backend Engineering Design patterns udemy course (link redirects to udemy with coupon) https://backend.husseinnasser.com

On March 7 2023, Loom users started seeing each others data as a result of cookies getting leaked from the CDN. This loom security breach is really critical. Let us discuss   0:00 Intro 1:00 Why Cookies 2:00 How this happens 5:50 What caused it? 7:30 How Loom solved it? 8:20 Reading the RCA 10:30 Remedies

Discord engineering goes into details of how they migrated from Cassandra to ScyllaDB, improved the performance of their reads and writes and rearchitected their backend to support the new load. It is an interesting episode lets get into it 0:00 Intro 1:50 Relational vs Distributed 7:00 The Cassandra Troubles 11:00 SnowFlake vs UUID 14:30 B+Tree 19:20 B+Tree and SSDs 25:30 LSM Trees 31:00 Hot partitions 36:00 Cassandra Garbage Collector Pauses 40:00 Changing the Architecture 45:00 The Data Services 55:00 The Migration 1:02:00 Zoned Named Spaces 1:04:00 Summary Article here How Discord Stores Trillions of Messages https://discord.com/blog/how-discord-stores-trillions-of-messages

Creating a listener on the backend application that accepts connections is simple. You listen on an address-port pair, connection attempts to that address and port will get added to an accept queue; The application accepts connections from the queue and start reading the data stream sent on the connection. However, what part of your application does the accepting and what part does the reading and what part does the execution? You can architect your application in many ways based on your use cases. I have a medium post just exploring the different options. In this video I explore the PostgreSQL process architecture in details. Please note that the information here is derived from both the Postgres doc and code. Discussions about scalability and performance are solely based on my opinions. 0:00 Intro 1:30 Overview 3:30 Postgres MVCC 5:30 Processes vs Threads 7:40 Postmaster Process 8:00 Backend Processes 13:30 Shared Buffers 14:52 Background Workers 17:18 Auxiliary Processes 17:45 Background Writer 22:30 Checkpointer 23:40 Logger 24:06 Autovacuum Launcher and Workers 25:30 WAL Processes 28:53 Startup Process Read full article https://medium.com/@hnasr/postgresql-process-architecture-f21e16459907

The Alt-Svc header/frame is a capability that allows the server to adverse alternative services to the connected application available in protocols, ports or domains. It is available as a response header alt-svc and also as an HTTP/2 frame. Let us discuss this capability. 0:00 Intro 1:38 what is alt-svc? 5:30 uses of h3 in alt-svc 8:00 alt-svc header 10:00 Alt-svc header with 103 early hints 14:48 h2 altsvc frame 18:30 SVCB DNS record 21:20 Summary Fundamentals of Backend Engineering Design patterns udemy course (link redirects to udemy with coupon) https://backend.husseinnasser.com

Correction: Google is implementing the proposal originally submitted by researchers from Georgia institute of tech. I incorrectly said in the video that google is proposing this . Google is finally implementing a proposal from 2008 by researchers from Georgia institute of technology to make DNS cache poisoning . https://astrolavos.gatech.edu/articles/increased_dns_resistance.pdf https://datatracker.ietf.org/doc/html/draft-vixie-dnsext-dns0x20-00 0:00 Intro 2:00 How DNS Work 5:00 DNS Cache Poisoning 14:00 gOoGLe dot CoM 16:20 ASCII 0x20 casing 18:30 Randomizing the casing with encryption 22:30 limitations of this proposal 24:00 Credits

https://dropbox.tech/infrastructure/increasing-magic-pocket-write-throughput-by-removing-our-ssd-cache-disks In this episode of the backend engineering show I’ll discuss how Dropbox improved their write through put by 20% by removing all their SSDs (yes I was surprised too). DropBox uses an SSD layer as a write-back cache with SMR drives as their backend persistent storage. They changed their model to write directly to the hard drives. 0:00 Intro 2:00 Article Summary 3:00 SMR Drives 6:00 SSD Cache & WriteBack 8:00 Replacing Cache 9:30 Storage Engine Background 14:30 Why did they do it 15:00 The limitation of SSDs & Zoned Namespaces 19:30 Updating the Storage Engine 22:30 Tradeoffs 26:00 Rollout 28:00 Summary

The communication between backend applications and database systems always fascinated me. The protocols keep evolving and we are in constant search for an efficient protocol that best fit the workload of Backend-DB communication. In this episode of the backend engineering show I go through a blog written by  @PlanetScale  doing an experimentation of using HTTP/3 and HTTP/2 comparing it with MySQL Binary protocol. https://planetscale.com/blog/faster-mysql-with-http3 0:00 Intro 7:45 MySQL Binary vs HTTP 10:20 The Tests 15:00 Connection Cost + Select 1 22:00 Parallel Select 26:00 The cost of H2 and H3

Fundamentals of Database Engineering udemy course (link redirects to udemy with coupon) https://database.husseinnasser.com Shopify posted a blog on tips to for scalable payment system, one tip peeked my interest related to switching from UUID to ULID. I explore the reasoning behind this in this video. https://shopify.engineering/building-resilient-payment-systems 0:00 Intro 1:30 idempotency 6:30 UUID vs ULID 9:50 Clustered Index 13:30 Why UUID4 Inserts are slow 17:15 How ULID helps Shopify 22:00 Problem with tail pages 25:00 Does ULID help in all cases? Fundamentals of Backend Engineering Design patterns udemy course (link redirects to udemy with coupon) https://backend.husseinnasser.com

I’m a big believer that database systems share similar core fundamentals at their storage layer and understanding them allows one to compare different DBMS objectively. For example, How documents are stored in MongoDB is no different from how MySQL or PostgreSQL store rows. Everything goes to disk, the trick is to fetch what you need from disk efficiently with as fewer I/Os as possible, the rest is API.  In this video I discuss the evolution of MongoDB internal architecture on how documents are stored and retrieved focusing on the index storage representation. I assume the reader is well versed with fundamentals of database engineering such as indexes, B+Trees, data files, WAL etc, you may pick up my database course to learn the skills. Let us get started. Fundamentals of Backend Engineering Design patterns udemy course (link redirects to udemy with coupon) https://backend.husseinnasser.com Fundamentals of Networking for Effective Backends udemy course (link redirects to udemy with coupon) https://network.husseinnasser.com Fundamentals of Database Engineering udemy course (link redirects to udemy with coupon) https://database.husseinnasser.com