Data Scale Challenges Start Early

59sec Snip

00:00

Play full episode

Summary

Transcript

Episode notes

Building a web application in its early stages often involves minimal user engagement, leading to relatively low operating costs. However, the need for substantial data to inform machine learning models arises even at this nascent phase. Developers encounter data scale challenges from the outset, rather than solely as user scale increases. This emphasizes that the foundation of AI solutions relies heavily on data pipelines, highlighting that effective prompt engineering and fine-tuning need to consider the vast data requirements from the very beginning of the application lifecycle.

LLMs are becoming more mature and accessible, and many teams are now integrating them into common business practices such as technical support bots, online real-time help, and other knowledge-base-related tasks. However, the high cost of maintaining AI teams and operating AI pipelines is becoming apparent.

Maxime Armstrong and Yuhan Luo are Software Engineers at Dagster, which is an open source platform for orchestrating data and AI pipelines. They join the show with Sean Falconer to talk about running cost-effective AI pipelines.

Sean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from information visualization to quantum computing. Currently, Sean is Head of Marketing and Developer Relations at Skyflow and host of the podcast Partially Redacted, a podcast about privacy and security engineering. You can connect with Sean on Twitter @seanfalconer.