14min chapter

Software Huddle cover image

Serverless Clickhouse with Tyler Wells

Software Huddle

CHAPTER

Evolution of Data Ingestion and Processing Methods

The chapter explores the evolution of data ingestion methods in Propel, discussing the initial emphasis on Snowflake integration, the development of additional methods like webhooks and S3 in Parquet format, and plans to support platforms like Kafka. It delves into Clickhouse's efficiency in handling data ingestion, discussing its capabilities with S3 and Parquet files, web hook strategies for handling JSON data, and the challenges faced in processing modern data infrastructure. The conversation also covers the importance of materialized views in Clickhouse, access policies for multi-tenant scenarios, and considerations around data pooling based on different use cases.

00:00

Get the Snipd
podcast app

Unlock the knowledge in podcasts with the podcast player of the future.
App store bannerPlay store banner

AI-powered
podcast player

Listen to all your favourite podcasts with AI-powered features

Discover
highlights

Listen to the best highlights from the podcasts you love and dive into the full episode

Save any
moment

Hear something you like? Tap your headphones to save it with AI-generated key takeaways

Share
& Export

Send highlights to Twitter, WhatsApp or export them to Notion, Readwise & more

AI-powered
podcast player

Listen to all your favourite podcasts with AI-powered features

Discover
highlights

Listen to the best highlights from the podcasts you love and dive into the full episode