A universal query engine in Rust (with Predrag Gruevski)
Feb 7, 2025
auto_awesome
Predrag Gruevski, creator of Trustfall, discusses his innovative work in developing a universal query engine in Rust. He explores the potential of querying diverse data sources, integrating REST APIs and databases with filesystems. The conversation dives into optimizing query performance and the challenges of building a flexible syntax for complex queries. Predrag also touches on the role of semantic versioning and linting tools like Cargo Semver Checks in maintaining code consistency. His insights reveal a vision for streamlined, efficient data interactions.
Trustfall offers a universal query engine in Rust that seamlessly integrates diverse data sources like APIs and databases into a single interface.
The architecture of Trustfall emphasizes minimizing execution time and data transfer costs by defining clear access policies during setup rather than on execution.
Trustfall's design allows developers to create specialized adapters for various data sources, focusing on query formulation and schema optimization rather than low-level implementations.
Deep dives
Understanding Queries Across Data Sources
SQL queries, REST queries, and GraphQL queries all serve a common purpose: retrieving data from various data sources. Even searching for files on a system can be classified as querying, highlighting the versatility of querying concepts. Code linting is also viewed through this lens, where source code can be treated as structured data needing analysis for error detection. This perspective opens the possibility of constructing a universal query engine capable of handling disparate data types from databases, APIs, and file systems seamlessly.
Introducing Trustfall: A Universal Query Engine
Trustfall allows the mixing of various data sources into a single interface, enabling users to craft complex queries without being tied to the specifics of each data source. It simplifies query construction by treating databases, APIs, or computational data as uniformly accessible. With its ability to execute queries over multiple environments, it drastically reduces friction when accessing live or dynamically generated data. This confluence of data sources not only streamlines processes but also empowers developers to explore and analyze data more efficiently.
Optimizing Query Performance and Cost
The design of Trustfall emphasizes minimizing both execution time and data transfer costs when performing queries across different sources. This is achieved by defining clear access policies during the setup of data sources rather than on each query execution. Trustfall's architecture allows automatic optimization to reduce unnecessary data fetching while promoting efficient querying strategies based on the configuration of the data source. Therefore, users interact with a simplified querying process without being burdened by the underlying complexities of performance tuning.
Implementing Trustfall for Querying
Trustfall is designed to facilitate the construction of specialized adapters for various data sources, making the querying process adaptable. Developers start by defining the queries they aim to execute, leading to a schema tailored to support those queries effectively. Trustfall provides tools to generate boilerplate code that ensures consistent implementations of the necessary adapter functions, significantly easing the development process. This approach allows developers to focus on formulating queries and optimizing the schema rather than getting bogged down with low-level details.
Transforming the Way We Query Data
Trustfall encourages a paradigm shift away from rigid database-centric querying toward a more flexible, inclusive approach. By blurring the lines between traditional data sources, it promotes creativity in querying practices, enabling developers to formulate innovative queries across heterogeneous datasets. The capability to integrate insights from various domains leads to more effective data analysis and problem-solving. Ultimately, Trustfall empowers developers to embrace a holistic view of their data ecosystem, redefining how they interact with and extract value from diverse data sources.
Trustfall is a library based on a simple question - what happens if we can query absolutely anything? If you could join REST APIs and databases with filesystems and dockerfiles? It’s possible in theory because those are all just datasources. Predrag Gruevski is trying to make it easy by building a universal query engine, with pluggable datasources, all in Rust.
This week we dive into Trustfall to figure out how it works. How do you model nearly anything as a datasource? How do you make it easy to extend? And what does it take to optimize a query that’s going to be spread out over multiple systems and potentially multiple servers? Questions, questions, questions - all about the act of asking our systems questions. 😃
How to Query Almost Everything: https://predr.ag/querying/
SemVer In Rust (talk): https://predr.ag/blog/semver-in-rust-tooling-breakage-and-edge-cases/#cargo-semver-checks-lints-are-database-queries-in-disguise
Support Developer Voices on Patreon: https://patreon.com/DeveloperVoices
Support Developer Voices on YouTube: https://www.youtube.com/@developervoices/join
Kris on Mastodon: http://mastodon.social/@krisajenkins
Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/
Kris on Twitter: https://twitter.com/krisajenkins
Get the Snipd podcast app
Unlock the knowledge in podcasts with the podcast player of the future.
AI-powered podcast player
Listen to all your favourite podcasts with AI-powered features
Discover highlights
Listen to the best highlights from the podcasts you love and dive into the full episode
Save any moment
Hear something you like? Tap your headphones to save it with AI-generated key takeaways
Share & Export
Send highlights to Twitter, WhatsApp or export them to Notion, Readwise & more
AI-powered podcast player
Listen to all your favourite podcasts with AI-powered features
Discover highlights
Listen to the best highlights from the podcasts you love and dive into the full episode