#267 The Developer Experience - How Do We Delight Data Developers? - Zhamak's Corner 30
Nov 9, 2023
auto_awesome
Zhamak and a developer discuss the challenges of data product development and the need for better tooling. They explore the evolution of data tools like Docker and Cloud Foundry, emphasizing the importance of cohesion and simplified management. They also draw lessons from the software industry, highlighting the benefits of abstracting infrastructure and providing opinionated frameworks for developers.
Automating away the challenges of coordinating across different tools and platforms can simplify data product development and improve the developer experience.
Bringing all the components of a data product together in a single interface eliminates the need for developers to juggle multiple tools and empowers them to focus on modeling data and ensuring version control.
Deep dives
Challenges in Data Product Development
A major challenge in data product development is the time and effort spent coordinating across different tools and jumping between them. This distracts from the actual development of the data product and creates unnecessary dependencies and lifecycle management issues. The focus should be on automating away these challenges and creating a cohesive experience, just like in software development. Examples such as Ruby on Rails and Cloud Foundry show how managing dependencies and containerization can improve the developer experience. Shifting to a domain-oriented approach and bundling all the disparate pieces together can simplify data product development, versioning, and lifecycle management.
The Importance of Cohesion in Data Products
Bringing all the components of a data product together is crucial for a cohesive and simplified developer experience. Having a single interface that manages the different tools and applications eliminates the need for developers to juggle multiple tools and platforms. By bundling everything together, including metadata, documentation, and code, developers can focus on modeling data, generating data, and ensuring version control. Automation of this complexity without sacrificing flexibility empowers developers to build and manage data products easily, without getting bogged down by the underlying infrastructure or needing to use multiple tools.
Lessons from Software Development
Software development offers valuable lessons for improving the data development experience. The concepts of two-factor apps and Cloud Foundry demonstrate the importance of abstracting infrastructure and managing dependencies. The focus in data development should be on solutions that enable developers to use the tools they prefer while providing a simple experience for managing complexity, versioning, and lifecycle management. Examples like Ruby on Rails and Heroku highlight how opinionated frameworks and containerization enhance the developer experience. To progress in the data management and analytics space, similar evolutions need to take place to move away from fragmented tools and towards a domain-oriented approach.
The current data product developer experience sucks. In software, we have one simple interface that manages all the pieces needed to get the job done, packages those components. In data, we have to jump through hoops and interfaces across many tools.
Right now, the developer has to manage everything themselves which is a ton of work. But it's also a big risk because of lifecycle management - if everything isn't packaged and deployed as one, you have dependencies drifting from each other.
There are many places from software we can learn from as to how to do this containerization. Ruby-on-Rails and CloudFoundry are good places to look.
Sponsored by NextData, Zhamak's company that is helping ease data product creation.