In this engaging discussion, Matthew Rocklin, the creator of Dask and co-founder of Coiled, and Nat Tabris, a staff software engineer at Coiled, delve into the complexities of running Python workloads at cloud scale. They unveil the reality of outgrowing local data processing, showcasing a live demo of spinning up a 1,000-core cluster. The pair shares savvy insights on choosing between pandas and Polars, optimizing costs, and the benefits of ephemeral clusters. Tune in for real-world lessons on navigating the cloud!
