Lessons in building and scaling data teams w/ Erik Bernhardsson, former CTO of Better.com
Dec 8, 2021
auto_awesome
Former CTO of Better.com and early Spotify manager, Erik Bernhardsson, discusses building recommendation systems, best practices for data team scaling, and the essence of music. Topics include managing data teams, programming languages, and measuring impact. Explore how to govern data teams effectively and embrace the right tools for success.
Building recommendation systems entails thoughtful considerations on construction necessity.
Data teams should be viewed as an engineering discipline, emphasizing programming language adoption.
Balanced centralization/decentralization models optimize resource allocation and foster team collaboration.
Deep dives
Working towards Better Data Operations and Collaboration
The importance of robust data operations and collaboration within the industry was discussed, emphasizing the need to advance data-driven practices. The episode highlighted the value of effective team structures and the exploration of innovative data ideas to drive industry progress.
Building Recommendation Systems and Data Team Management
Delving into the intricacies of building recommendation systems, the conversation touched upon the considerations of whether to construct these systems and how data teams should be conceptualized as an engineering discipline. Insights were shared on managing and governing data teams, measuring impact, embracing programming languages in data operations, and addressing data abundance within companies.
Navigating Career Pathways in Data and Technical Evolution
The episode shed light on the evolution of technical expertise, discussing career trajectories and the shifts in coding languages over time. Insights were shared about the experiences of building recommendation systems and managing engineering teams at renowned companies like Spotify and Better.com, highlighting the importance of technical adaptability and exploring new ideas in the data space.
Balancing Centralization and Decentralization in Data Teams
Exploring the challenges of centralization and decentralization in data team structures, the episode underscored the significance of finding a hybrid model that optimizes resource allocation and fosters collaboration. It highlighted the drawbacks of extreme centralization or decentralization, emphasizing the need for a balanced approach to drive efficient data operations and team effectiveness.
Importance of Full-Stack Engineering Teams
The podcast explored the concept of restructuring engineering teams to have full-stack engineers rather than specialized engineers. It highlighted that having engineers with diverse skill sets who can handle front-end, back-end, and mobile tasks can lead to more efficient and effective outcomes. By having team members who can work on various tasks well, it eliminates the need for multiple layers of specialization and allows for better collaboration within teams, under the leadership of individuals who are both engineering managers and tech leads.
Challenges and Evolution in Data Engineering
The episode discussed challenges and the evolving nature of data engineering. It emphasized the comparison between the maturity of software engineering tools and the constant changes in the data engineering landscape. The podcast touched on how the field of data engineering is still relatively new, resulting in a 'wild west' environment with new tools emerging frequently. Additionally, it explored the need for reevaluation in incorporating software engineering principles into the data field and pondered on potential new languages or environments that could improve data engineering workflows.
In today’s episode, I talk with Erik Bernhardsson, the former CTO at Better.com and early engineering manager at Spotify. We talked about how to build recommendation systems (including whether you should build them at all). We also covered some best practices on building and scaling data teams, what the essence of music is, and whether data teams should be thought of as an engineering discipline (and what programming languages we should embrace in the data world). We also took a deep dive into how to manage and govern the data team and how to measure the impact.
Also - if you're looking to stay up to date on the best and most colorful data industry commentary between episodes, check out our new newsletter: The Operational Analytics Diggest.
