Data Engineering Podcast Fast Analytics On Semi-Structured And Structured Data In The Cloud
Oct 8, 2019
54:39
Summary
The process of exposing your data through a SQL interface has many possible pathways, each with their own complications and tradeoffs. One of the recent options is Rockset, a serverless platform for fast SQL analytics on semi-structured and structured data. In this episode CEO Venkat Venkataramani and SVP of Product Shruti Bhat explain the origins of Rockset, how it is architected to allow for fast and flexible SQL analytics on your data, and how their serverless platform can save you the time and effort of implementing portions of your own infrastructure.
Announcements
- Hello and welcome to the Data Engineering Podcast, the show about modern data management
- When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With 200Gbit private networking, scalable shared block storage, and a 40Gbit public network, you’ve got everything you need to run a fast, reliable, and bullet-proof data platform. If you need global distribution, they’ve got that covered too with world-wide datacenters including new ones in Toronto and Mumbai. And for your machine learning workloads, they just announced dedicated CPU instances. Go to dataengineeringpodcast.com/linode today to get a $20 credit and launch a new server in under a minute. And don’t forget to thank them for their continued support of this show!
- This week’s episode is also sponsored by Datacoral. They provide an AWS-native, serverless, data infrastructure that installs in your VPC. Datacoral helps data engineers build and manage the flow of data pipelines without having to manage any infrastructure. Datacoral’s customers report that their data engineers are able to spend 80% of their work time invested in data transformations, rather than pipeline maintenance. Raghu Murthy, founder and CEO of Datacoral built data infrastructures at Yahoo! and Facebook, scaling from mere terabytes to petabytes of analytic data. He started Datacoral with the goal to make SQL the universal data programming language. Visit Datacoral.com today to find out more.
- You listen to this show to learn and stay up to date with what’s happening in databases, streaming platforms, big data, and everything else you need to know about modern data management. For even more opportunities to meet, listen, and learn from your peers you don’t want to miss out on this year’s conference season. We have partnered with organizations such as O’Reilly Media, Dataversity, Corinium Global Intelligence, Alluxio, and Data Council. Upcoming events include the combined events of the Data Architecture Summit and Graphorum, the Data Orchestration Summit, and Data Council in NYC. Go to dataengineeringpodcast.com/conferences to learn more about these and other events, and take advantage of our partner discounts to save money when you register today.
- Your host is Tobias Macey and today I’m interviewing Shruti Bhat and Venkat Venkataramani about Rockset, a serverless platform for enabling fast SQL queries across all of your data
Interview
- Introduction
- How did you get involved in the area of data management?
- Can you start by describing what Rockset is and your motivation for creating it?
- What are some of the use cases that it enables which would otherwise be impractical or intractable?
- How does Rockset fit into the infrastructure and workflow of data teams and what portions of a typical stack does it replace?
- Can you describe how the Rockset platform is architected and how it has evolved as you onboard more customers?
- Can you describe the flow of a piece of data as it traverses the full lifecycle in Rockset?
- How is your storage backend implemented to allow for speed and flexibility in the query layer?
- How does it manage distribution, balancing, and durability of the data?
- What are your strategies for handling node and region failure in the cloud?
- You have a whitepaper describing your architecture as being oriented around microservices on Kubernetes in order to be cloud agnostic. How do you handle the case where customers have data sources that span multiple cloud providers or regions and the latency that can result?
- How is the query engine structured to allow for optimizing so many different query types (e.g. search, graph, timeseries, etc.)?
- With Rockset handling a large portion of the underlying infrastructure work that a data engineer might be involved with, what are some ways that you have seen them use the time that they have gained and how has that benefitted the organizations that they work for?
- What are some of the most interesting/unexpected/innovative ways that you have seen Rockset used?
- When is Rockset the wrong choice for a given project?
- What have you found to be the most challenging and the most exciting aspects of building the Rockset platform and company?
- What do you have planned for the future of Rockset?
Contact Info
- Venkat
- Shruti
- @shrutibhat on Twitter
Parting Question
- From your perspective, what is the biggest gap in the tooling or technology for data management today?
Closing Announcements
- Thank you for listening! Don’t forget to check out our other show, the Data Engineering Podcast for the latest on modern data management.
- Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
- If you’ve learned something or tried out a project from the show then tell us about it! Email hosts@podcastinit.com) with your story.
- To help other people find the show please leave a review on iTunes and tell your friends and co-workers
- Join the community in the new Zulip chat workspace at pythonpodcast.com/chat
Links
- Rockset
- Oracle
- VMWare
- Rube Goldberg Machine
- SnowflakeDB
- Protocol Buffers
- Spark
- Presto
- Apache Kafka
- RocksDB
- InnoDB
- Lucene
- Log Structured Merge Tree (LSTM)
- Kubernetes
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA