#10 – Matt Wonlaw: cr-sqlite, syncing strategies and incremental view maintenance
Jun 12, 2024
auto_awesome
Matt Wonlaw, a local-first tool builder, discusses CRDTs, SQLite, and incremental view maintenance. Topics include software dev for submarines, event sourcing, CR SQLite, challenges of SQLite in browsers, Materialite, and data modeling for multiplayer apps.
The challenges of maintaining reactivity in SQLite and optimizing IndexedDB integration were significant hurdles in local-first software development.
Efforts to optimize query result maintenance focused on efficiently updating data in response to user actions, aiming to enhance performance and user interaction.
The limitations of SQLite for modern interactive applications led to exploration of alternative projects like Materialize, bringing differential data flow to JavaScript for reactive query building.
Deep dives
Challenges in Developing Reactive Databases
In the quest to enhance local-first software, challenges arose in developing reactive databases with SQLite in the browser. Handling extensions, transactions, and updates posed significant hurdles. Issues such as transforming C code into Rust to improve performance, synchronizing transactions, and optimizing IndexedDB integration were key obstacles. Maintaining reactivity in SQLite, ruling out memory-intensive operations, and navigating transaction controls were among the complex challenges faced.
Optimizing Query Results Maintenance
Efforts were made to optimize the maintenance of query results for richer user interaction. Addressing the need for incremental upkeep of queries, the focus shifted to efficiently update data in response to user actions without reprocessing queries from scratch. This entailed investigating methods to selectively invalidate and patch affected query results, aiming to enhance performance and responsiveness within the app.
Transitioning Away from SQLite for Enhanced Interactivity
The limitations of SQLite's request-response design for modern interactive applications became apparent. The constraints in handling reactive scenarios and incremental updates led to a gradual transition away from SQLite. Recognizing the mismatch between SQLite's architecture and the needs of reactive local-first software, exploration into alternative projects and data flow mechanisms emerged, marking a shift in the approach towards achieving enhanced interactivity and responsiveness.
Materialize: Bringing Differential Data Flow to JavaScript
Materialize was born out of the need to bring differential data flow to JavaScript, enabling the composition of filter, map, reduce, and join operators for reactive query building. This allowed for the creation of complex queries updated reactively upon any write events, reminiscent of SQL's select, where functions. Materialize aimed to mirror SQL concepts in JavaScript, offering functionalities like map for selecting columns, filter for condition application, reduce for aggregate functions, and concat for stream merging.
Replicash, Reflect, and Local First Advancements
The discussion expanded to various tools like Replicash, SIR SQLite, Yjs, AutoMerge, Electric SQL, and PowerSync, highlighting the differences between CRDT and rebasing models for syncing data in distributed systems. CRDTs offer message order independence but require a single authoritative server, while rebasing models like Replicash allow for server-controlled merges. The recommendation to experiment with different technologies to gauge compatibility was emphasized, emphasizing the importance of choosing the right tool based on use case considerations.
The guest of this episode is Matt Wonlaw, a prolific local-first tool builder who’s behind projects such as Vlcn, cr-sqlite and Materialite. Most recently Matt also joined Rocicorp to work on their new product. This conversation will go deep on his projects covering CRDTs, SQLite and incremental view maintenance.
