In this podcast episode, we talked with Adrian Brudaru about the past, present and future of data engineering.

About the speaker:

Adrian Brudaru studied economics in Romania but soon got bored with how creative the industry was, and chose to go instead for the more factual side. He ended up in Berlin at the age of 25 and started a role as a business analyst. At the age of 30, he had enough of startups and decided to join a corporation, but quickly found out that it did not provide the challenge he wanted.

As going back to startups was not a desirable option either, he decided to postpone his decision by taking freelance work and has never looked back since. Five years later, he co-founded a company in the data space to try new things. This company is also looking to release open source tools to help democratize data engineering.

0:00 Introduction to DataTalks.Club

1:05 Discussing trends in data engineering with Adrian

2:03 Adrian's background and journey into data engineering

5:04 Growth and updates on Adrian's company, DLT Hub

9:05 Challenges and specialization in data engineering today

13:00 Opportunities for data engineers entering the field

15:00 The "Modern Data Stack" and its evolution

17:25 Emerging trends: AI integration and Iceberg technology

27:40 DuckDB and the emergence of portable, cost-effective data stacks

32:14 The rise and impact of dbt in data engineering

34:08 Alternatives to dbt: SQLMesh and others

35:25 Workflow orchestration tools: Airflow, Dagster, Prefect, and GitHub Actions

37:20 Audience questions: Career focus in data roles and AI engineering overlaps

39:00

The role of semantics in data and AI workflows

41:11 Focusing on learning concepts over tools when entering the field

45:15 Transitioning from backend to data engineering: challenges and opportunities

47:48 Current state of the data engineering job market in Europe and beyond

49:05 Introduction to Apache Iceberg, Delta, and Hudi file formats

50:40 Suitability of these formats for batch and streaming workloads

52:29 Tools for streaming: Kafka, SQS, and related trends

58:07 Building AI agents and enabling intelligent data applications

59:09Closing discussion on the place of tools like DBT in the ecosystem

🔗 CONNECT WITH ADRIAN BRUDARU

Linkedin - / data-team Website - https://adrian.brudaru.com/ 🔗 CONNECT WITH DataTalksClub

Join the community - https://datatalks.club/slack.html Subscribe to our Google calendar to have all our events in your calendar - https://calendar.google.com/calendar/... Check other upcoming events - https://lu.ma/dtc-events LinkedIn - /datatalks-club Twitter - /datatalksclub Website - https://datatalks.club/

‘SNL,’ Kamala’s laugh, Hannibal Lecter: Culture and the 2024 election