The Backend Engineering Show with Hussein Nasser

How Discord Stores Trillions of Messages - A deep dive

Mar 11, 2023

01:09:20

Creator website

Highlights

AI Chapters

Episode notes

What Happens When You Update an SSD?

03:13

How to Store Twelfths of Messages in 2017

03:22

Is Cassandra a Good Data Service?

03:27

Cilla DB Migration

02:19

Introduction

3min

How to Store Twelfths of Messages in 2017

3min

MongoDB Cassandra Messages

3min

CQL Statements - How to Cluster Messages in a Database

3min

You You ID Is a Snowflake

4min

Cassandra Reads Are More Expensive Than Right?

2min

How to Insert a Row in a Clustered MySQL Database

3min

What Happens When You Update an SSD?

3min

What Goes Really Bad When You Have an Erasable Unit?

4min

10.

Write Write Writes Are Extremely Fast

2min

11.

How Bad Be the Street Can Affect Your SSDs

2min

12.

Can You Hotspot a Partition?

3min

13.

Cassandra Cluster Maintenance Problem

3min

14.

Cassandra

4min

15.

Cilla DB Migration

2min

16.

Cilla DB

4min

17.

Is Cassandra a Good Data Service?

3min

18.

Using a Load Balancing to Consolidate Requests

3min

19.

The Data Services Are Moving to Cassandra

2min

20.

How to Migrate Messages Into the Cassandra Cluster?

3min

21.

Getting Value Faster Isn't It?

4min

22.

NVMe

3min

23.

I'm Not Defending Cassandra

2min

24.

Is Silla DB a Better Architecture?

2min

Discord engineering goes into details of how they migrated from Cassandra to ScyllaDB, improved the performance of their reads and writes and rearchitected their backend to support the new load. It is an interesting episode lets get into it

0:00 Intro

1:50 Relational vs Distributed

7:00 The Cassandra Troubles

11:00 SnowFlake vs UUID

14:30 B+Tree

19:20 B+Tree and SSDs

25:30 LSM Trees

31:00 Hot partitions

36:00 Cassandra Garbage Collector Pauses

40:00 Changing the Architecture

45:00 The Data Services

55:00 The Migration

1:02:00 Zoned Named Spaces

1:04:00 Summary

Article here How Discord Stores Trillions of Messages

https://discord.com/blog/how-discord-stores-trillions-of-messages

Remember Everything You Learn from Podcasts

Save insights instantly, chat with episodes, and build lasting knowledge - all powered by AI.