Learn how Spotify scaled their engineering team without losing speed and independence, creating a unified developer platform called Backstage. Discover the challenges faced, including silos and communication gaps, and how they were overcome through transparency and collaboration. Hear how the development of Backstage with a plugin model transformed the engineering culture and enabled scalability.
Spotify consolidated its CI infrastructure in just six months, solving the complexity and duplication challenges while addressing risks to security and scalability.
The development of Backstage, a centralized developer portal, played a crucial role in transforming Spotify's engineering culture by providing transparency, reducing fragmentation, and empowering engineers to access information and collaborate effectively.
Deep dives
Spotify faced challenges in preparing for IPO
In 2016, Spotify needed to prepare for an IPO, requiring careful planning and clean internal operations. However, they discovered vulnerabilities in their CI infrastructure, with over 200 bespoke Jenkinses maintained by 300 teams. This decentralized approach led to complexity and duplication, posing a risk to security. To address this, Pia Nelson and her team consolidated the CI infrastructure in just six months, not only solving the CI challenge but also leading a transformation in Spotify's engineering practices.
Pia's experience joining Spotify and the need for consolidation
When Pia joined Spotify, she was struck by the unique team identities and autonomous culture. However, as she started working on the CI project, it became clear that consolidation was necessary. The challenge was not only technical but also cultural and business-related, as maintaining 300 individual paths to production with varying states posed risks to security and scalability. Pia's goal was to consolidate the CI infrastructure while preserving team identities and belonging.
The shift towards team ownership and collaboration
As Pia and her team worked on consolidating Spotify's CI infrastructure, they realized the importance of team ownership and collaboration. They embarked on a mission to transform the engineering culture by shifting towards team ownership of adoption and embracing collaboration. By addressing challenges such as rumor-driven development and lack of transparency, they created a culture focused on impact and speed. This shift also aligned with Spotify's emphasis on autonomy and team identity, fostering a sense of empowerment and belonging.
The role of backstage in fostering collaboration and transparency
One of the key initiatives in Spotify's transformation was the development of backstage, a centralized developer portal. Backstage enabled teams to visualize the connection between components and owners, eliminating the need for rumor-driven development. It provided transparency, reduced fragmentation, and empowered engineers to access information and collaborate more effectively. Backstage's success led to its open-sourcing, ensuring that Spotify remains a leader in speed and development while fostering a collaborative and transparent culture.
How did Spotify scale from 10 engineers to 100s to 1000s ...without slowing down? Without becoming corporate?
Facing an IPO deadline, Pia Nilsson worked with 300 teams to transform how Spotify built software. She spearheaded a movement that led them from working in silos to a unified developer platform.
Hear the inside story of how Spotify's Platform teams embraced transparency and customer focus to create Backstage — now used by companies worldwide.
It's an amazing tale of ingenuity and perseverance. Hear Spotify's secret to scaling engineering without losing speed and independence. Don't miss it!