The Developer Experience Survey at Peloton was designed to gather qualitative and quantitative insights to understand the onboarding experience and shape solutions for engineers joining the company.
The survey design process at Peloton involved extensive research, both externally and internally, including studying how other companies approach surveys and creating a 'day in the life of a developer' journey map to design relevant survey questions.
The importance of designing actionable survey questions, engaging executive leadership, and continuously iterating the survey to keep it relevant and effective were highlighted.
Deep dives
Building the Tech Learning Function at Peloton
When Vangja Sadataram joined Peloton, her mission was to build the tech learning function. As part of this, she looked at the onboarding process for engineers and realized that numbers alone did not tell the whole story. She conducted over 100 interviews to gather qualitative and quantitative insights to understand the onboarding experience and shape the right solutions for all engineers joining Peloton.
The Inspiration for the Developer Experience Survey
The inspiration for the Developer Experience Survey at Peloton came from discussions with leadership, including Vangja's former boss Jason. They wanted to understand the developer experience qualitatively and focus on more than just metrics. The vision for a comprehensive survey to drive developer efficiency, effectiveness, and engagement was already present in the leadership team's goals.
Designing and Refining the Survey
The survey design process at Peloton involved extensive research, both externally and internally. They studied how other companies approach surveys and leveraged the space framework for guidance. Internally, they created a 'day in the life of a developer' journey map to understand the current experience and desired outcomes. This mapping helped in designing the survey questions and factors that captured the key aspects of the developer experience at Peloton.
Designing an Actionable Survey
The speaker emphasizes the importance of designing survey questions that are actionable and able to drive action. They highlight the challenge of making questions actionable and the need to partner with platform engineering teams and product managers to gather relevant insights. The focus is on creating a survey that is not too long and doesn't cause survey fatigue for participants. The speaker also mentions the importance of continuously iterating the survey to keep it relevant and effective.
Presenting Survey Results to Executives
The speaker discusses how they presented the survey results to executive leadership. They shared a detailed report with the entire organization and followed it up with a one-page summary highlighting key findings. The speaker emphasizes the importance of engaging executive leadership and getting their feedback and buy-in. They also mention the importance of communicating the results to different teams and aligning with talent partners to avoid duplication. The process of analyzing the survey data, identifying key areas of improvement, and developing action plans is also highlighted.
Thansha Sadacharam, who leads Tech Learning and Insights at Peloton walks us through the journey of building the company’s developer experience survey. She shares what went into the survey’s design, rollout, and maintenance, as well as the different teams involved.
Discussion points:
(1:19) Where the idea for running a developer survey originated
(6:36) Advice for other leaders getting buy-in for these initiatives
(11:27) The first steps in designing the survey
(18:21) How the survey incorporated benchmarking
(20:30) Measuring developer satisfaction
(22:37) Refining the question items
(25:50) How long the survey was
(26:50) What was involved in trimming the questions
(29:28) Writing survey questions
(33:12) How much time was spent developing the survey
(35:19) The communication plan for launching the survey
(42:05) Driving participation rates
(45:21) Sampling and how often surveys are being sent