Eric Anderson (@ericmander) and Frank McSherry (@frankmcsherry) dive into Materialize, a source-available streaming database that lets engineers build real-time applications. Frank is a data processing expert whose work at Microsoft Research on the Timely and Differential Dataflow models culminated in the Materialize project. Tune in to today’s episode to learn how the team at Materialize are making the technology from cutting-edge data research accessible to a wider swath of users. In this episode we discuss: Sharing early ideas with an “academic open source” approach How Materialize made a commitment to correctness Frank’s developmental philosophy of iterative thinking Novel applications for the Materialize community Changing the way we approach problems with real-time data processing Links: Materialize Naiad: A Timely Dataflow System DryadLINQ Apache Arrow People mentioned: Arjun Narayan (@narayanarjun) Derek Murray (@mrry)

Eric Anderson (@ericmander) speaks with Thomas Graf (@tgraf__) about Cilium, the open-source networking, observability, and security software for cloud-native applications based on eBPF. Thomas is the co-founder and CTO of Isovalent, which maintains both eBPF and Cilium. Listen to today’s episode for a discussion of how Thomas’ work has leveled up the Linux kernel and the possibilities of network infrastructure in a cloud-native world. In this episode we discuss: The impact of simultaneous development on Cilium and eBPF Google’s incorporation of Cilium Shortening the gap between writing kernel code and its deployment What JavaScript and eBPF have in common Cilium’s sister project, Hubble Links: Cilium eBPF Isovalent Red Hat OpenShift Kubernetes Docker New GKE Dataplane V2 increases security and visibility for containers SPIFFE Istio People mentioned: Brendan Gregg (@brendangregg) Other episodes: Istio on Contributor

Eric Anderson (@ericmander) and Jeremiah Lowin (@jlowin) discuss Prefect, a workflow management system and data orchestration tool under development as an open-source project. Jeremiah initially created Prefect to solve a technical challenge specific to his own work, but soon realized that it was appealing to a very wide range of different clients. Listen to today’s episode to learn why Jeremiah believes most attempts to build a unified framework for solving data orchestration fail. In this episode we discuss: Solving the “negative engineering problem” Learning from the complaints of data engineers at Apache Airflow The difficulty of having a product that serves two masters How COVID changed the direction of Prefect Links: Prefect Apache Airflow Why Not Airflow? People mentioned: Jim O'Shaughnessy (@jposhaughnessy) Patrick O’Shaughnessy (@patrick_oshag)

Eric Anderson (@ericmander) catches up with Torin Sandall (@sometorin), co-creator of Open Policy Agent (OPA), the open-source, general-purpose policy engine. By focusing on demonstrating OPA’s value through case studies, targeted interviews, and word-of-mouth, Torin and the folks at Styra were able to grow OPA into the emerging standard for unified policy enforcement across the cloud-native stack. In this episode we discuss: When Netflix stumbled across OPA and delivered its “Cinderella moment” Why OPA was designed to be developer-centric The value of demonstrating OPA’s use cases to the industry How one user created an RPG engine with OPA Links: Open Policy Agent Styra OpenStack LinkerD Hacker News Kubernetes KubeCon OPA Gatekeeper conftest Corrupting the Open Policy Agent to Run My Games Envoy Styra Academy People mentioned: Tim Hinrichs (@tlhinrchs) William Morgan (@wm) Kevin Hoffman (@kevinhoffman) Other episodes: LinkerD on Contributor Envoy on Contributor

Eric Anderson (@ericmander) and Maxim Fateev (@mfateev) trace the development of Temporal, an open-source workflow orchestration engine. At Uber, Maxim co-created the project’s predecessor, Cadence, but Temporal’s roots stretch farther back to include lessons learned at Amazon and Microsoft. In this episode, learn how 18 years of experience in asynchronous messaging and workflows culminated in the foundation of Temporal. In this episode we discuss: Why Maxim quit Uber to start his own company Differences between Temporal and Cadence How Uber is filling the position that Google once had incubating open-source projects Maxim’s advice for aspiring open-source founders Related Links: Temporal Cadence Kafka HashiCorp BanzaiCloud Hacker News Andreesen Horowitz TChannel Hadoop People mentioned: Samar Abbas (@samarabbas77)

Eric Anderson (@ericmander) and Manish Jain (@manishrjain) discuss the impact of Dgraph, an open-source database with a graph backend that Manish describes as “a search engine acting as a database.” Manish took a gamble when he chose GraphQL as his project’s query language shortly after its release by Facebook in 2015. Now, GraphQL has grown immensely in popularity and the bet has paid off, as Dgraph leads the cutting edge of databases in this new space. Make sure to check out the Dgraph team’s conference, “GraphQL In Space,” which will be held virtually on September 10th at graphqlcon.space. In this episode we discuss: How Manish was ahead of the curve at Google The chance circumstances in the Australian job market that led to Dgraph Building trust between open-source developers and their community Why the Dgraph team decided to hold their upcoming conference “In Space” The future of databases and GraphQL Related Links: Dgraph GraphQL In Space GraphQL Badger MongoDB BigTable Cassandra Spanner Elasticsearch People mentioned: Scott Kelly (@StationCDRKelly)

Eric Anderson (@ericmander) talks to Martin Traverso (@mtraverso), Dain Sundstrom (@daindumb) and David Phillips (@electrum32) about their collaboration on Presto, an open-source distributed SQL query engine for big data. The three engineers worked together at three different companies before deciding to solve an efficiency problem for data analytics at Facebook in 2012. Listen to today’s episode to learn about the careful planning and technical philosophy behind the development and design of Presto. In this episode we discuss: Starting an open-source project at Facebook in the early 2010s The importance of making Presto “dirt simple to install” What is “documentation driven development” Bootstrapping the growth of an open-source community How a single query caused a brownout across Facebook infrastructure Related Links: Presto Starburst Ning Netezza ProofPoint Hadoop Postgres Hive OpenCompute @Scale Arm Treasure Data Qubole People mentioned: Jay Parikh (@jayparikh)

Nathan Killoran (@co9olguy) guides Eric Anderson (@ericmander) through the cutting-edge world of quantum machine learning at Xanadu, a quantum computing company that is innovating with its use of photonics. Nathan is Xanadu’s Head of Software, Algorithms, & Quantum Machine Learning, and has detailed insight on their main open-source software projects, StrawberryFields and PennyLane. On today’s episode, Nathan explains how the barrier to contributing may be lower than you think, even if you don’t have a PhD in quantum physics. In this episode we discuss: Designing software for Xanadu’s unique approach to quantum computing Machine learning, differentiable programming and more in the quantum domain How even high school students can contribute to an open-source quantum computing project Is there a road map for quantum machine learning? Nathan’s “blue sky” interview questions Links: Xanadu StrawberryFields PennyLane ProjectQ TensorFlow Quantum PyTorch Qiskit Pyquil Cirq Alpine Quantum Technologies Quantum Open Source Foundation Unitary Fund People mentioned: Christian Weedbrook, CEO of Xanadu (@_cweedbrook)

Eric Anderson (@ericmander) talks to Alexey Milovidov (@alexey-milovidov) and Ivan Blinkov (@blinkov) about their work on Clickhouse, an open source analytical database from the team at Yandex. Originally designed to support Yandex.Metrica, word of this powerful tool spread rapidly inside the company, and the idea was hatched to make Clickhouse into a truly open source project. Tune in to learn about how Alexey petitioned management to accept what initially seemed like a “crazy” idea - and how the risk paid off. In this episode we discuss: Differences between Clickhouse and similar products Why some open source projects are more successful than others The history of open source at Yandex What makes a good open source developer Building an international community Links: Clickhouse Yandex.Metrica Altinity Postgres Oracle Infobright InfinityDB MongoDB Vertica Dremel: Interactive Analysis of Web-Scale Datasets (2010) CatBoost BEM Presto Druid Greenplum Apache Spark

Eric Anderson (@ericmander) chats with William Morgan (@wm), CEO of Buoyant and a creator of the open source service mesh, LinkerD. As a former infrastructure engineer at Twitter, William leveraged his experience there to help develop what would become effectively the first service mesh. Listen to today’s episode to find out how the team at Buoyant originally coined the term, and are continuing to define the concept today. In this episode we discuss: Pioneering the very first service mesh Why Buoyant rejected the open core model How the industry is shifting away from the “nights and weekends” community Rewriting LinkerD from scratch Links: LinkerD Buoyant Dive Kubernetes Docker Finagle HAProxy NGINX CNCF Prometheus Cisco Webex Istio