Tyler Wells, CTO and co-founder at Propel, discusses building data platforms at Twilio and Skype. Topics include real-time analytics challenges, Propel's solutions, infrastructure development insights, S3's importance, and the impact of AI on software engineering.
Propel focuses on efficient data ingestion methods like Snowflake integration, webhooks, and S3 Parquet format for real-time analytics.
Compute costs are managed as the highest expense, emphasizing readiness and responsiveness while prioritizing stability and performance over scaling intricacies.
Propel's usage-based billing model evolves to offer competitive rates, reflecting added value, and infrastructure management costs for customer flexibility.
Deep dives
Ingestion Methods at Propel
Propel offers various ingestion methods, including direct integration with Snowflake, webhooks for real-time data ingestion, and support for S3 Parquet format. The company initially focused on Snowflake integration due to its reputation as a gold standard for data warehousing. Webhooks allow for immediate data insertion, but throttling can be applied based on request rates. Their approach with S3 Parquet format leverages Clickhouse's native support for this format, ensuring efficient data ingestion and storage.
Compute Utilization and Infrastructure Costs
Propel manages compute costs as the highest expense, necessitating low but constant utilization for readiness and responsiveness to varying query loads. While reserved instances help offset costs, maintaining availability 24/7 remains crucial. The company prioritizes stability and performance over scaling intricacies, ensuring customer data integrity and responsiveness.
Usage-Based Billing Model Evolution
Propel's usage-based billing model initially met a mixed response from customers, with some unfamiliar with the model pondering worst-case cost scenarios. The company's pricing model underwent adjustments to offer competitive rates reflecting added value and infrastructure management costs. Changes such as reduced storage pricing and introducing pricing tiers for dedicated instances aim to enhance customer understanding and flexibility.
Data Ingestion and Processing
Propel efficiently ingests data through direct methods like Snowflake integration and webhooks, as well as support for formats like S3 Parquet. Clickhouse's native support facilitates smooth data ingestion from various sources, enabling real-time processing and analysis. The system adapts to different data sources and processing needs, continually evolving to meet changing customer requirements and use cases.
Adapting to Real-World Data Challenges
In the podcast, it is discussed how in the real world, customers often do not follow the preferred pipe-building process, necessitating a shift towards giving customers more control over joining and cleaning data. This change reflects a trend where organizations use multiple infrastructure pieces for analytics and applications, leading to a desire for more comprehensive solutions at a smaller scale.
Enhancing Data Security and Ease of Use
The conversation also delves into topics like multi-tenant access policies using Jot tokens for API authentication and enforcing tenancy-specific data access. This approach aims to simplify data access for end-users while maintaining security measures. Additionally, advancements like automated Jot token creation for various languages showcase the efficiency gains AI brings to tasks like coding and data access control.
Today's episode is with Tyler Wells. Tyler is the CTO and co-founder at Propel. He was an early employee at Skype (and Microsoft after the acquisition) as well as Twilio. While at Twilio, Tyler helped build a data platform to power customer-facing analytics for a major Twilio feature. Propel is the productized version of that for other teams looking to build similar experiences.
In this episode, we see how this real-time, flexible analytics problem is tricky for a lot of teams, as well as how Propel is helping to solve the problem. We also cover some of Alex's favorite hobby horses for infrastructure developers -- what it's like building infrastructure services, how to think about billing, how S3 is becoming ubiquitous, and what to do about cross-AZ network costs.
Timestamps
02:29 Introduction
08:05 What is Propel?
22:28 ClickHouse
29:15 Target Customers
30:28 Billing Model
35:10 S3 becoming a key part?
36:47 Cross AZ Network Costs
41:56 Current Support
51:39 Access Policies
55:39 Rapid Fire
01:03:16 AI replacing Software Engineers?
Show Notes
Data Chaos Podcast
https://www.propeldata.com/
Remember Everything You Learn from Podcasts
Save insights instantly, chat with episodes, and build lasting knowledge - all powered by AI.
