Dive into the thrilling rise of a new social media platform as it blossoms from a stagnant base to a staggering 30 million users. Hear insights from a software engineer on how they managed explosive growth without breaking the bank. Discover the intricacies of the Personal Data Server and the innovative strategies for incident management during challenging periods. The conversation is packed with anecdotes about tech misadventures, video streaming hurdles, and the playful side of building resilient infrastructure in the digital age.
01:05:57
forum Ask episode
web_stories AI Snips
view_agenda Chapters
menu_book Books
auto_awesome Transcript
info_circle Episode notes
question_answer ANECDOTE
Explosive Growth
Blue Sky grew from 100,000 users to nearly 30 million in 18 months.
This rapid growth felt like a decade and required significant infrastructure scaling.
insights INSIGHT
Tiered Infrastructure
Blue Sky uses three infrastructure tiers: singleton services in cloud, core data services on-prem, and PDSs on bare metal.
This approach balances cost, performance, and scalability for different needs.
volunteer_activism ADVICE
Cloud vs. On-Prem Storage
Commoditized cloud products like block storage offer good value up to a point.
Consider cost savings and management overhead when scaling storage beyond several petabytes.
Get the Snipd Podcast app to discover more snips from this episode
The Big Ideas Behind Reliable, Scalable, and Maintainable Systems
Martin Kleppman
This book helps software engineers and architects understand the pros and cons of various technologies for storing and processing data. It covers fundamental principles, trade-offs, and design decisions in data systems, including scalability, consistency, reliability, efficiency, and maintainability. The book delves into distributed systems research, the architecture of data systems, and how to make informed decisions about different tools and technologies. It does not provide detailed instructions on specific software packages but focuses on the underlying principles and trade-offs essential for designing data-intensive applications[2][4][5].
Bluesky has been on a roller coaster of growth for over a year. From the early days of figuring out a new distributed social protocol—AT protocol—to actually building it and inviting 30 million of their closest friends. Not only has the site gone through tremendous growth, the team has been optimizing, re-architecting, and adding features the entire time.
Jaz is a software engineer focused on the infrastructure at Bluesky, and they share how they achieved exponential growth without exponential costs. We cover some of the key components of the protocol and how that affects the architecture.
There’s some amazing advice from the trenches we know you’ll enjoy.
Show Highlights (0:00) Intro (5:00) Jaz’s background (12:30) Bluesky Infrastructure (17:00) Predicting the future (20:00) What is a PDS? (22:30) Relay and firehose (26:00) Work queues (30:00) Scaling physical servers (37:00) How do you handle incidents? (41:00) Where’s Kubernetes? (43:30) How video changes (45:00) Data locality (46:30) Hardware decisions (53:00) What bad decisions? (57:00) Launching video (1:00:00) What’s next?
About Jaz
Jaz is a software engineer who learned from on-the-job experience. They have a background with hardware which makes them better with software. If they’re not drinking Monster they’re building a single purpose database, or maybe they’re doing both. Jaz went from building with AT protocol to building AT protocol in a matter of months. They also have an impressive collection of plushies and power tools.
Fork Around And Find Out 