Graph Databases In Production At Scale Using DGraph with Manish Jain - Episode 44
Aug 20, 2018
auto_awesome
Manish Jain, Creator of DGraph, discusses the benefits of storing and querying data as a graph, how DGraph overcomes limitations, building a distributed, consistent database, and the use case of integrating 51 data silos into a single database cluster.
D-Graph is a distributed graph database designed to scale out and provide modern database functionalities, focusing on real-time transactional use cases and providing horizontal scalability.
D-Graph is well-suited for various use cases where graph-based data structures and relationships play a vital role, such as recommendation systems, fraud detection, ad tech, and customer 360 scenarios, but may not be the best choice for purely flat time series data.
Deep dives
D-Graph: A Scalable Distributed Graph Database
D-Graph is a distributed graph database designed to scale out and provide modern database functionalities. The motivation behind building D-Graph was the need for a fast and scalable graph database. Existing solutions were limited to single-server performance and did not meet the requirements of modern data sets. The recent explosion in the use of graph-oriented storage systems can be attributed to the increasing complexity and interconnectedness of data. D-Graph stands out by focusing on real-time transactional use cases and providing horizontal scalability. It aims to be the most advanced cloud database in the market, offering consistent replication, transactional capabilities, and efficient query processing. While D-Graph is a CP system, it strives to provide high performance and low-latency experiences for developers.
Common Use Cases and Viable Alternatives
D-Graph is well-suited for various use cases where graph-based data structures and relationships play a vital role. One common use case is recommendation systems, where graph databases excel in providing personalized recommendations based on complex data connections. Fraud detection is another area where D-Graph is being successfully utilized, allowing real-time analysis of interconnected data to identify potential fraudulent activities. Ad tech companies use D-Graph to deliver targeted ads based on user activities and preferences. Additionally, D-Graph is valuable in customer 360 scenarios, where it unifies disparate data sources to provide insights into customer behaviors and interactions across various touchpoints. While D-Graph is a powerful tool for graph-based workloads, it may not be the best choice for purely flat time series data, for which specialized time series databases may be more suitable.
Architecture and Unique Features
D-Graph's architecture has gone through several iterations to achieve its current stable state. It started with a single-server design and evolved into a distributed graph database with sharding capabilities. Replication based on Raft consensus ensures data consistency and availability. One of D-Graph's unique design decisions is to shard data based on edges rather than nodes, enabling more efficient and lower-latency queries by minimizing network calls. With its flexible schema approach, D-Graph offers both schema-on-write and schema-on-read capabilities, providing developers with iteration flexibility. Underneath, D-Graph utilizes a key-value store and optimizes data representation using sophisticated techniques such as posting lists for efficient graph traversals and intersections. Querying is performed using a modified version of GraphQL that preserves relationship information and returns subgraphs rather than mere lists of results.
Deployment, Scalability, and Future Directions
Deploying and operating a D-Graph cluster is relatively straightforward, and it offers scalability by adding more servers to distribute the data and workload. D-Graph's architecture allows the cluster to scale horizontally, and there are no inherent limitations on cluster size. However, visual monitoring and management tools are still being developed, and there is room for improvement in terms of operational tooling. D-Graph's current focus is on stabilizing the database, addressing user requests, and integrating requested features into the product roadmap. Future plans include enhancing security features and expanding access control capabilities. Additionally, D-Graph aims to continue building a strong and supportive community, encouraging users to provide feedback, file issues, and contribute to the project's ongoing development.
The way that you store your data can have a huge impact on the ways that it can be practically used. For a substantial number of use cases, the optimal format for storing and querying that information is as a graph, however databases architected around that use case have historically been difficult to use at scale or for serving fast, distributed queries. In this episode Manish Jain explains how DGraph is overcoming those limitations, how the project got started, and how you can start using it today. He also discusses the various cases where a graph storage layer is beneficial, and when you would be better off using something else. In addition he talks about the challenges of building a distributed, consistent database and the tradeoffs that were made to make DGraph a reality.
Preamble
Hello and welcome to the Data Engineering Podcast, the show about modern data management
When you’re ready to build your next pipeline you’ll need somewhere to deploy it, so check out Linode. With private networking, shared block storage, node balancers, and a 40Gbit network, all controlled by a brand new API you’ve got everything you need to run a bullet-proof data platform. Go to dataengineeringpodcast.com/linode to get a $20 credit and launch a new server in under a minute.
If you have ever wished that you could use the same tools for versioning and distributing your data that you use for your software then you owe it to yourself to check out what the fine folks at Quilt Data have built. Quilt is an open source platform for building a sane workflow around your data that works for your whole team, including version history, metatdata management, and flexible hosting. Stop by their booth at JupyterCon in New York City on August 22nd through the 24th to say Hi and tell them that the Data Engineering Podcast sent you! After that, keep an eye on the AWS marketplace for a pre-packaged version of Quilt for Teams to deploy into your own environment and stop fighting with your data.
Python has quickly become one of the most widely used languages by both data engineers and data scientists, letting everyone on your team understand each other more easily. However, it can be tough learning it when you’re just starting out. Luckily, there’s an easy way to get involved. Written by MIT lecturer Ana Bell and published by Manning Publications, Get Programming: Learn to code with Python is the perfect way to get started working with Python. Ana’s experience
as a teacher of Python really shines through, as you get hands-on with the language without being drowned in confusing jargon or theory. Filled with practical examples and step-by-step lessons to take on, Get Programming is perfect for people who just want to get stuck in with Python. Get your copy of the book with a special 40% discount for Data Engineering Podcast listeners by going to dataengineeringpodcast.com/get-programming and use the discount code PodInit40!
Go to dataengineeringpodcast.com to subscribe to the show, sign up for the mailing list, read the show notes, and get in touch.
Your host is Tobias Macey and today I’m interviewing Manish Jain about DGraph, a low latency, high throughput, native and distributed graph database.
Interview
Introduction
How did you get involved in the area of data management?
What is DGraph and what motivated you to build it?
Graph databases and graph algorithms have been part of the computing landscape for decades. What has changed in recent years to allow for the current proliferation of graph oriented storage systems?
The graph space is becoming crowded in recent years. How does DGraph compare to the current set of offerings?
What are some of the common uses of graph storage systems?
What are some potential uses that are often overlooked?
There are a few ways that graph structures and properties can be implemented, including the ability to store data in the vertices connecting nodes and the structures that can be contained within the nodes themselves. How is information represented in DGraph and what are the tradeoffs in the approach that you chose?
How does the query interface and data storage in DGraph differ from other options?
What are your opinions on the graph query languages that have been adopted by other storages systems, such as Gremlin, Cypher, and GSQL?
How is DGraph architected and how has that architecture evolved from when it first started?
How do you balance the speed and agility of schema on read with the additional application complexity that is required, as opposed to schema on write?
In your documentation you contend that DGraph is a viable replacement for RDBMS-oriented primary storage systems. What are the switching costs for someone looking to make that transition?
What are the limitations of DGraph in terms of scalability or usability?
Where does it fall along the axes of the CAP theorem?
For someone who is interested in building on top of DGraph and deploying it to production, what does their workflow and operational overhead look like?
What have been the most challenging aspects of building and growing the DGraph project and community?
What are some of the most interesting or unexpected uses of DGraph that you are aware of?
