Scaling Graph Analytics Without ETL: Inside PuppyGraph’s Architecture
Data Engineering Podcast
00:00
Outro
Tobias closes the episode, invites listeners to subscribe, and gives show administration details.
Play episode from 53:40
Transcript
Transcript
Episode notes
Summary
In this episode Weimo Liu, co‑founder of PuppyGraph, talks about the engineering behind their “zero-copy” graph querying engine for lakehouse and database sources. He explores how PuppyGraph lets you run Cypher and Gremlin traversals and graph algorithms directly on data in Iceberg, Delta, Hudi, Hive, and even MongoDB—without loading into a separate graph store. Weimo explains their edge-sharded, vectorized, MPP architecture that tackles hub nodes, multi-hop traversals, and shuffle at scale, targeting sub-second to single-digit-second workloads. He digs into practical graph data modeling on top of normalized and denormalized tables, logical views, and flexible mappings; strategies for caching, adaptive reads, and leveraging Iceberg metadata; and how PuppyGraph’s operator-based engine unifies query and algorithms. He also covers real-world applications—from cybersecurity log analysis to entity resolution and agentic workflows—when to choose embedded or transactional graph databases instead, and what’s next for enterprise features and broader warehouse integrations.
Announcements
Parting Question
%
In this episode Weimo Liu, co‑founder of PuppyGraph, talks about the engineering behind their “zero-copy” graph querying engine for lakehouse and database sources. He explores how PuppyGraph lets you run Cypher and Gremlin traversals and graph algorithms directly on data in Iceberg, Delta, Hudi, Hive, and even MongoDB—without loading into a separate graph store. Weimo explains their edge-sharded, vectorized, MPP architecture that tackles hub nodes, multi-hop traversals, and shuffle at scale, targeting sub-second to single-digit-second workloads. He digs into practical graph data modeling on top of normalized and denormalized tables, logical views, and flexible mappings; strategies for caching, adaptive reads, and leveraging Iceberg metadata; and how PuppyGraph’s operator-based engine unifies query and algorithms. He also covers real-world applications—from cybersecurity log analysis to entity resolution and agentic workflows—when to choose embedded or transactional graph databases instead, and what’s next for enterprise features and broader warehouse integrations.
Announcements
- Hello and welcome to the Data Engineering Podcast, the show about modern data management
- This episode is sponsored by DataDriven.io, the free data engineering interview prep platform built by data engineers for data engineers. Ever walked into a data engineering interview and gotten a question that has nothing to do with real data engineering work? Interviewing is its own skill, separate from the job. Watch your code execute live, inspect Spark internals, and whiteboard your data models and pipelines and defend your decisions. Unlike SQL-only or Python-only practice, DataDriven.io covers the full interview loop: star schemas, slowly changing dimensions, grain and fact table design, idempotency, watermarks, dead letter queues, change data capture, and backpressure. Every question comes from real Data Engineer interview loops at Google, Amazon, Meta, Stripe, Databricks, Netflix, and Airbnb. Go to dataengineeringpodcast.com/datadriven today to start practicing.
- Your host is Tobias Macey and today I'm interviewing Weimo Liu about the engineering behind PuppyGraph's zero-copy ETL for querying your lakehouse as a graph
- Introduction
- How did you get involved in the area of data management?
- Can you start by describing what PuppyGraph is and the story behind it?
- What are some of the key use cases that people are turning to PuppyGraph and graph data models for?
- Graph engines have struggled to take off for several years, not least of which is due to the difficulty of scaling them to large data volumes as a result of the topological nature of the data. Can you describe the architecture of PuppyGraph and some of the ways that you are addressing that challenge of data volume for graphs?
- latency/data exploration
- types of traversals and limitations
- lakehouse architecture pros/cons for graphs
- data modeling/translation
- shortcomings of zero-ETL and how transforming the underlying representation could provide benefits
- For someone who is looking for a graph engine to support a connected data use case, what are the guiding questions that you would ask to lead them toward PuppyGraph vs. a dedicated graph database like Memgraph/Neo4J/etc.?
- What are the most interesting, innovative, or unexpected ways that you have seen PuppyGraph used?
- What are the most interesting, unexpected, or challenging lessons that you have learned while working on PuppyGraph?
- When is PuppyGraph the wrong choice?
- What do you have planned for the future of PuppyGraph and graph data exploration on large data volumes?
Parting Question
- From your perspective, what is the biggest gap in the tooling or technology for data management today?
- Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The AI Engineering Podcast is your guide to the fast-moving world of building AI systems.
- Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
- If you've learned something or tried out a project from the show then tell us about it! Email [email protected] with your story.
- PuppyGraph
- TigerGraph
- Google F1
- Graph Database
- Google Pregel
- Iceberg
- Graph Supernode
- MPP == Massively Parallel Processing
- Spark GraphX
- Trino
- Ladybug DB
- lance-graph
- KuzuDB
- MemGraph
- Labelled Property Graph
- RDF Triples
- Cypher Query Language
- Gremlin
- CDC == Change Data Capture
- Neo4J
- JanusGraph
- NetworkX
- PyTorch
- DuckDB
- Iceberg Array
- LanceDB
- Palo Alto Networks
- Columnar ADBC
%
The AI-powered Podcast Player
Save insights by tapping your headphones, chat with episodes, discover the best highlights - and more!
