Apache Spark is a popular open source analytics engine for large-scale data processing. Applications can be written in Java, Scala, Python, R, and SQL. These applications have flexible options to run on like Kubernetes or in the cloud.

The company Data Mechanics is a cloud-native Spark platform for data engineers. It runs continuously optimized Apache Spark workloads on a managed Kubernetes cluster within the user’s cloud account. They boast a 50%-75% cost reduction from cloud providers by dynamically scaling applications based on load and automatically tuning app configurations based on the historical Spark pipeline runs. Their Kubernetes clusters are deployed within user accounts so user data never leaves the environment and they handle the cluster management.

In this episode we talk to Jean-Yves Stephan, Co-Founder and CEO at Data Mechanics. Jean-Yves previously worked as a Software Engineer then a Tech Lead Manager at Databricks. We discuss big data engineering in Spark and the unique advantages of using Data Mechanics to make Spark development easier and more cost effective.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Data Mechanics: Data Engineering with Jean-Yves Stephan appeared first on Software Engineering Daily.