Loki: A Scale Problem

Loki is a new way of storing logs. It's based on key value pairs that identify a log stream, which is the sort of unique thing that we store. We bundle up those streams into blocks of a certain size and then flush them in an object store. The indexes in Loki are so small because it's just some key value pairs. You can set how much parallelism you want to read at the same time. And you now store a really small index that's easy to load in and out of memory.