The Backend Engineering Show with Hussein Nasser

Explore the six stages of becoming a better software engineer. Discover how deep understanding transforms your relationship with technology. Learn the art of articulating technical concepts without jargon. Delve into the emotional journey of facing inherent limitations in technology. Gain insights into software efficiency through the lens of operating systems. Ultimately, appreciate the beauty in both the flaws and strengths of technology as you navigate your path in engineering.

A new Linux kernel patch promises to enhance network performance by implementing zero-copy techniques. Instead of the traditional method, this update turns data reading into a notification process, allowing userspace to directly access the data's location. The conversation dives into the intricacies of memory management and how this innovation could revolutionize data transmission efficiency in operating systems. Curious about its impact on performance? They break down the benefits and implications of this groundbreaking approach.

Discover how Cloudflare revolutionized its CDN caching with a global cache purge that executes in under 150 ms. The shift from a centralized lazy purge to a decentralized active system showcases remarkable engineering feats. Explore the intricacies of RockDB, peer-to-peer data structures, and how these innovations address previous limitations. Delve into the performance implications of LSM versus BTree and unpack the benefits of immediate purge requests. This deep dive reveals the complexities and clever strategies behind cutting-edge backend engineering.

Fundamentals of Database Engineering udemy course https://databases.win MySQL has been having bumpy journey since 2018 with the release of the version 8.0. Critical crashes that made to the final product, significant performance regressions, and tons of stability and bugs issues. In this video I explore what happened to MySql, are these issues getting fixed? And what is the current state of MySQL at the end of 2024. 0:00 Intro 2:00 MySQL 8.0 vs 5.7 Performance 11:00 Critical Crash in 8.0.38, 8.4.1 and 9.0.0 15:40 Is 8.4 better than 8.0.36? 16:30 More Features = More Bugs 22:30 Summary and my thoughts resources https://x.com/MarkCallaghanDB/status/1786428909376164263 https://www.percona.com/blog/do-not-upgrade-to-any-version-of-mysql-after-8-0-37/ http://smalldatum.blogspot.com/2024/09/mysql-innodb-vs-sysbench-on-large-server.html https://www.percona.com/blog/mysql-8-0-vs-5-7-are-the-newer-versions-more-problematic/

Fundamentals of Operating Systems Course https://oscourse.win In this video I use strace a performance tool that measures how many system calls does a process makes. We compare a simple task of reading from a file, and we run the program in different runtimes, namely nodejs, buns , python and native C. We discuss the cost of kernel mode switches, system calls and pe 0:00 Intro 5:00 Code Explanation 6:30 Python 9:30 NodeJS 12:30 BunJS 13:12 C 16:00 Summary

Fundamentals of Operating Systems Course https://os.husseinnasser.com When do you use threads? I would say in scenarios where the task is either 1) IO blocking task 2) CPU heavy 3) Large volume of small tasks In any of the cases above, it is favorable to offload the task to a thread. 1) IO blocking task When you read from or write to disk, depending on how you do it and the kernel interface you used, the write might be blocking. This means the process that executes the IO will not be allowed to execute any more code until the write/read completes. That is why you see most logging operations are done on a secondary thread (like libuv that Node uses) this way the thread is blocked but the main process/thread can resume its work. If you can do file reads/writes asynchronously with say io_uring then you technically don't need threading. Now notice how I said file IO because it is different than socket IO which is always done asynchronously with epoll/select etc. 2) CPU heavy The second use case is when the task requires lots of CPU time, which then starves/blocks the rest of the process from doing its normal job. So offloading that task to a thread so that it runs on a different core can allow the main process to continue running on its the original core. 3) Large volume of small tasks The third use case is when you have large amount of small tasks and single process can't deliver as much throughput. An example would be accepting connections, a single process can only accept connections so fast, to increase the throughput in case where you have massive amount of clients connecting, you would spin multiple threads to accept those connections and of course read and process requests. Perhaps you would also enable port reuse so that you avoid accept mutex locking. Keep in mind threads come with challenges and problems so when it is not required. 0:00 Intro 1:40 What are threads? 7:10 IO blocking Tasks 17:30 CPU Intensive Tasks 22:00 Large volume of small tasks

Dive into the intriguing world of timeouts in programming! Discover how they manage waiting periods, free up resources, and even fend off potential DOS attacks. Learn about different types of timeouts like connection and response, and their broader applications beyond just request processing. Engaging discussions highlight best practices for optimizing client-server interactions and ensuring efficient request handling. Prepare for unexpected insights that could change how you think about frontend and backend communications!

Learn more about database and OS internals, check out my courses  Fundamentals of database engineering https://databases.win  Fundamentals of operating systems https://oscourse.win This new PostgreSQL 17 feature is game changer. You see, postgres like most databases work with fixed size pages. Pretty much everything is in this format, indexes, table data, etc. Those pages are 8K in size, each page will have the rows, or index tuples and a fixed header. The pages are just bytes in files and they are read and cached in the buffer pool. To read page 0, for example, you would call read on offset 0 for 8192 bytes, To read page 1 that is another read system call from offset 8193 for 8192, page 7 is offset 57,345 for 8192 and so on.  If table is 100 pages stored a file, to do a full table scan, we would be making 100 system calls, each system call had an overhead (I talk about all of that in my OS course).  The enhancement in Postgres 17 is to combine I/Os you can specify how much IO to combine, so technically while possible you can scan that entire table in one system call doesn’t mean its always a good idea of course and Ill talk about that.  This also seems to included a vectorized I/O, with preadv system call which takes an array of offsets and lengths for random reads.  The challenge will become how to not read too much, say I’m doing a seq scan to find something, I read page 0 and found it and quit I don’t need to read any more pages. With this feature I might read 10 pages in one I/O and pull all its content, put in shared buffers only to find my result in the first page (essentially wasting disk bandwidth, memory etc)  It is going to be interesting to balance this out. 

Fundamentals of Operating Systems Course https://os.husseinnasser.com Why Windows Kernel connects slower than Linux I explore the behavior of TCP/IP stack in Windows kernel when it receives a RST from the backend server especially when the host is available but the port we are trying to connect to is not. This behavior is exacerbated by having both IPv6 and IPv4 and if the happy eye ball protocol is in place where IPv6 is favorable. 0:00 Intro 0:30 Fundamentals TCP/IP 3:00 Unreachable Port Behavior 6:00 Client Kernel Behavior (Linux vs Windows) 11:40 Slow TCP Connect on Windows 15:00 localhost, IPv6 and IPv4 20:00 Happy Eyeballs 28:00 Registry keys to change the behavior 31:00 Port Unreachable vs Host Unreachable https://daniel.haxx.se/blog/2024/08/14/slow-tcp-connect-on-windows/

In this episode of the backend engineering show I describe an interesting bug I ran into where the web server ran out of ephemeral ports causing the system to halt.  0:00 Intro 0:30 System architecture  2:20 The behavior of the bug 4:00 Backend Troubleshooting 7:00 The cause 15:30 Ephemeral ports on loopback