Data Engineering Podcast Behind The Scenes Of The Linode Object Storage Service
Mar 23, 2020
35:53
Summary
There are a number of platforms available for object storage, including self-managed open source projects. But what goes on behind the scenes of the companies that run these systems at scale so you don’t have to? In this episode Will Smith shares the journey that he and his team at Linode recently completed to bring a fast and reliable S3 compatible object storage to production for your benefit. He discusses the challenges of running object storage for public usage, some of the interesting ways that it was stress tested internally, and the lessons that he learned along the way.
Announcements
- Hello and welcome to the Data Engineering Podcast, the show about modern data management
- When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With 200Gbit private networking, scalable shared block storage, a 40Gbit public network, fast object storage, and a brand new managed Kubernetes platform, you’ve got everything you need to run a fast, reliable, and bullet-proof data platform. And for your machine learning workloads, they’ve got dedicated CPU and GPU instances. Go to dataengineeringpodcast.com/linode today to get a $20 credit and launch a new server in under a minute. And don’t forget to thank them for their continued support of this show!
- You listen to this show to learn and stay up to date with what’s happening in databases, streaming platforms, big data, and everything else you need to know about modern data management. For even more opportunities to meet, listen, and learn from your peers you don’t want to miss out on this year’s conference season. We have partnered with organizations such as O’Reilly Media, Corinium Global Intelligence, ODSC, and Data Council. Go to dataengineeringpodcast.com/conferences to learn more about these and other events, and take advantage of our partner discounts to save money when you register today.
- Your host is Tobias Macey and today I’m interviewing Will Smith about his work on building object storage for the Linode cloud platform
Interview
- Introduction
- How did you get involved in the area of data management?
- Can you start by giving an overview of the current state of your object storage product?
- What was the motivating factor for building and managing your own object storage system rather than building an integration with another offering such as Wasabi or Backblaze?
- What is the scale and scope of usage that you had to design for?
- Can you describe how your platform is implemented?
- What was your criteria for deciding whether to use an available platform such as Ceph or MinIO vs building your own from scratch?
- How have your initial assumptions about the operability and maintainability of your installation been challenged or updated since it has been released to the public?
- What have been the biggest challenges that you have faced in designing and deploying a system that can meet the scale and reliability requirements of Linode?
- What are the most important capabilities for the underlying hardware that you are running on?
- What supporting systems and tools are you using to manage the availability and durability of your object storage?
- How did you approach the rollout of Linode’s object storage to gain the confidence that you needed to feel comfortable with full scale usage?
- What are some of the benefits that you have gained internally at Linode from having an object storage system available to your product teams?
- What are your thoughts on the state of the S3 API as a de facto standard for object storage?
- What is your main focus now that object storage is being rolled out to more data centers?
Contact Info
Parting Question
- From your perspective, what is the biggest gap in the tooling or technology for data management today?
Links
- Linode Object Storage
- Xen Hypervisor
- KVM (Linux Kernel Virtual Machine)
- Linode API V4
- Ceph Distributed Filesystem
- Wasabi
- Backblaze
- MinIO
- CERN Ceph Scaling Paper
- RADOS Gateway
- OpenResty
- Lua
- Prometheus
- Linode Managed Kubernetes
- Ceph Swift Protocol
- Ceph Bug Tracker
- Linode Dashboard Application Source Code
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA