Apache Beam with Kenneth Knowles and Pablo Estrada

10 snips

Apr 6, 2022

Ask episode

AI Snips

Chapters

Transcript

Episode notes

ANECDOTE

Beam's Origin

Cloud Dataflow's open-source library allowed pipeline execution locally or on the cloud.
Other companies adapted this for Flink and Spark, leading to Apache Beam's creation.

INSIGHT

Beam's Value Proposition

Apache Beam's unified batch and streaming model, along with its emphasis on correctness, attracted developers.
Its rigorous semantics, especially in streaming, were unique and valuable.

ANECDOTE

Microservices vs. Frameworks

Microservices for data processing become obsolete as needs grow more complex.
Frameworks like Beam address this by handling execution and time semantics.

Get the Snipd Podcast app to discover more snips from this episode

Get the app

On the podcast this week, your hosts Stephanie Wong and Mark Mirchandani talk about the data processing tool Apache Beam with guests Pablo Estrada and Kenneth Knowles.

Kenn starts us off with an overview of how Apache Beam began and how Cloud Dataflow was involved. The unique batch and stream method and emphasis on correctness garnered support from developers early on and continues to attract users. Pablo helps us understand why Beam is a better option for certain projects looking to process large amounts of data. Our guests describe how Beam may be a better fit than microservices that could become obsolete as company needs change.

Next, we step back and take a look at why batch and stream is the gold standard of data processing because of its balance between low latency and ease of “being done” with data collection. Beam’s focus on the correctness of data and correctness in processing that data is a core component. With good data, processing becomes easier, more reliable, and cheaper. Kenn gives examples of how things can go wrong with bad data processing. Beam strives for the perfect combination of low latency, correct data, and affordability. Users can choose where to run Beam pipelines, from other Apache software offerings to Dataflow, which means excellent flexibility. Our guests talk about the pros and cons of some of these options and we hear examples of how companies are using Beam along with supporting software to solve data processing challenges.

To get started with Beam, check out Beam College or attend Beam Summit 2022.

Kenneth Knowles

Kenn Knowles is chair of the Apache Beam Project Management Committee. Kenn has been working on Google Cloud Dataflow—Google’s Beam backend—since 2014. Kenn holds a PhD in programming languages from the University of California, Santa Cruz.

Pablo Estrada

Pablo is a Software Engineer at Google, and a management committee member for Apache Beam. Pablo is big into working on an open source project, and has worked all across the Apache Beam stack.

Cool things of the week

Under the sea: Building the world’s fiber optic internet video
- Discovering Data Centers videos
Google Data Cloud Summit site
It’s official—Google Distributed Cloud Edge is generally available blog
- GCP Podcast Episode 228: Fastly with Tyler McMullen podcast
Save big by temporarily suspending unneeded Compute Engine VMs—now GA blog

Interview

Apache Beam site
Apache Beam Documentation site
Dataflow site
Apache Flink site
Apache Spark site
Apache Samza site
Apache Nemo site
Spanner site
BigQuery site
Beam College site
Beam College on Github site
Beam Developer Mailing List email
Beam User Mailing List email
Beam Summit site

What’s something cool you’re working on?

Mark is working on a new Apache Beam video series Getting Started Wtih Apache Beam

Hosts

Stephanie Wong and Mark Mirchandani