4min chapter

#106 - Moving on from the Modern Data Stack w/ Sarah Catanzaro

Monday Morning Data Chat

CHAPTER

Are There Tools That Make Data Modeling Easier?

I'm interested to see what happens from a formal practice in terms of interactive data modeling for example because it seems like it's a much different beast, then the way we've done things before. There really aren't that many I see that are widely used there's data vault which I think is gaining popularity but I don't. At the same time it might be a lot more effort than I would see a lot of teams wanting to invest in that practice and so. Yeah exactly. Well then I think to you bring up kind of data rich applications for example right interactive apps which I'm assuming has some sort of a real time element to it or at least a faster element than just once a

00:00

Transcript

Episode notes

Speaker 2

So what is the thing that's controversial why would why do people not think that

Speaker 1

that should be part of the algebra. So a lot of people so one of the things that we we spent a lot of time doing was optimizing for this ability to transpose and to kind of treat rows and columns interchangeably we actually built a system that could handle hundreds of billions of columns and there just aren't that many systems in the world that can handle that many columns just because like you have to have metadata for all of your columns in most systems and we had a way basically of kind of having the metadata be inferred at runtime which is which is another another key aspect of the data frames but having these kind of like short but really really fat or wide tables is not something that's extremely common in the kind of world of databases and so that's that's kind of why it ended up being like okay the controversial if you would find this image you would find

Speaker 2

this machine learning though 100%

Speaker 1

yeah 100% yes it's extremely common in machine learning but databases don't like that's changing these days because databases are starting to do a lot more but databases have historically not been well optimized for doing machine learning workloads and so this idea of the transpose and the same world I think is yeah I mean databases would just crash if you try to throw throw a hundred billion columns into them and

Speaker 2

you