121: Materialize Origins: Breaking Down Data Flow Layers with Arjun Narayan and Frank McSherry
The Data Stack Show
00:00
How to Parallelize Differential Data Flow
The real value comes from like, I mean, obviously you want to parallelize that. The reason differential data flow would want you to do it is because they automatically incrementalize as well. So these operators that we've forced you to use joins and reduces maps, filters, stuff like that, caused you to trick you into writing your program in an automatically incrementalizable form. You just won't be delighted either by its parallelization or by its incrementalization.
Transcript
Play full episode
