Preston Doster, an infrastructure architect at Twilio, discusses the intricacies of running Hachyderm, a Mastodon server with over 55,000 accounts. He shares compelling stories about scaling infrastructure and the challenges of moderation within the decentralized social media space. The conversation delves into funding strategies and the integration of tools like CDNs and Terraform for efficient management. Doster also highlights the legal complexities around content hosting and the parallels between engineering and music creation, offering a refreshing perspective on tech and creativity.
The transition of Sentry's metrics system highlights the importance of integrated data for effective debugging and developer support.
Decentralized social networks like Mastodon promote user connection and personalization, reminiscent of early internet experiences but raise moderation challenges.
Hackaderm's evolution from a basement server to a robust architecture illustrates the necessity of scalability and transparency amid rapid user growth.
Deep dives
Trace Connected Metrics
The discussion highlights the evolution of Sentry’s approach to metrics, moving from a traditional metrics product to a trace-connected metrics system. This transformation was driven by the realization that disconnected metrics do not effectively support the needs of developers, particularly in debugging contexts. By changing their APIs and focusing on creating a more integrated experience, Sentry has aimed to provide richer data that relates closely to the underlying code structure. This strategic shift is seen as a crucial investment, promising better insights and a smoother debugging process for users.
Decentralized Social Networking
The conversation explores the complexities and advantages of decentralized social networks, with a focus on ActivityPub and Mastodon. The hosts express enthusiasm for the decentralized aspects of the internet, noting how Mastodon allows users to connect in a manner reminiscent of the early web. They draw parallels to web rings, emphasizing the personalization and discovery aspects of social connections that are not as prevalent in today’s mainstream platforms. This element of decentralization raises questions about moderation and the user experience on distributed networks.
Infrastructure of Hackaderm
The infrastructure of Hackaderm, one of the largest Mastodon instances, is discussed, detailing its rapid growth following the Twitter exodus. The CTO explains how Hackaderm transitioned from a single basement server to a more robust architecture, which includes dedicated hardware in a data center. As user numbers surged from around 500 to over 30,000, the technical team adopted open-source software to manage scaling challenges and maintain decentralized operations. They emphasize the importance of transparency and open communication around server performance and how they handle user growth.
Moderation in the Fediverse
The episode delves into the challenges of moderating a decentralized network like Mastodon, where moderation practices significantly impact the user experience. Moderators have the ability to block individual users and even entire servers to maintain community standards. The hosts underscore the collective responsibility of moderation and the delicate balance required in handling potentially harmful content that may originate from other servers. Transparency in moderation practices is highlighted as essential for building trust within the community.
Future Considerations and Scalability
Future risks associated with operating a decentralized social network are addressed, particularly concerning data storage costs and legal compliance across jurisdictions. The team at Hackaderm is keenly aware of the implications of keeping a growing amount of user-generated content and the necessity of archiving policies as storage expenses rise. They also discuss the need for readiness against possible political pressures that could affect server operations or content moderation. Overall, proactive planning and community engagement are deemed vital for Hackaderm's sustained stability and growth.
Changelog++ members save 10 minutes on this episode because they made the ads disappear. Join today!
Sponsors:
Sentry – Code breaks, fix it faster. Use the code CHANGELOG when you sign up to get $100 off the team plan. Don’t just observe. Take action. Sentry is the only app monitoring platform built for developers that gets to the root cause for every issue. 100,000+ growing teams use sentry to find problems fast.
System Initiative – The future of DevOps automation (is here!) — System Initiative is an intuitive, powerful, collaborative replacement for Infrastructure as Code (IaC). The free tier is awesome (no credit card required) and you can get started in 3 clicks.
Timescale – Purpose-built performance for AI Build RAG, search, and AI agents on the cloud and with PostgreSQL and purpose-built extensions for AI: pgvector, pgvectorscale, and pgai.
One correction about the cost to run Hachyderm. Preston said it’s about $600/mo and after the interview emailed me to let me know it’s closer to $1000/mo