Overcoming Observability Challenges in Serverless ETL Pipelines

In this episode from AWS Summit Bengaluru, Poonam Pratik, Director at The Line Tech UK and AWS Community Builder, discusses practical approaches to serverless ETL implementation. We cover key aspects of data processing including quality control, AWS Glue orchestration, and effective data validation methods. Poonam explains how serverless architectures can reduce operational complexity while maintaining data accuracy. The conversation includes concrete examples of data partitioning, error notification systems, and observability practices. We conclude with a look at how AI and ML are changing data pipeline development.

With Poonam Pratik Patel, Director, The Line Tech, UK and AWS Community Builder

Getting started with serverless ETL on AWS Glue