Measuring and rolling out AI coding assistants | Eirini Kalliamvakou (GitHub)
Jan 31, 2024
auto_awesome
Eirini Kalliamvakou, a staff researcher at GitHub, talks about how AI coding assistance impacts developer productivity. They discuss recent research on AI tools, key findings from the study, and how to advocate for AI tools. They also explore the implications of AI on measuring productivity and the impact of AI on developer experience.
AI coding assistance, such as GitHub co-pilot, has shown strong correlations with improved productivity aspects like staying in the flow and completing tasks faster.
A study found that professional developers using co-pilot achieved a 55% faster completion time and higher success rate, with less experienced programmers benefiting more.
Deep dives
Research on AI coding assistance impact
Recent research evaluates how AI coding assistance, specifically GitHub co-pilot, impacts developer productivity. The study focuses on data gathered from evaluations to understand the effects of co-pilot usage by developers, teams, and organizations. The research aims to inform decision-making, investments, and the communication of effects and benefits of AI tools. One significant study shows a correlation between acceptance rate of co-pilot and various productivity aspects, such as staying in the flow, job fulfillment, and completing tasks faster.
Deep dive into the co-pilot study
A large-scale survey and interviews were conducted for the co-pilot study, aiming to understand user experiences and measure productivity effects. The survey questions were designed based on the space framework of productivity, covering dimensions like activity, satisfaction, efficiency, and flow. Correlations were analyzed between survey responses and telemetry data on co-pilot acceptance rates. The results showed strong correlations between acceptance rates and reported productivity improvements, including faster completion, flow, fulfillment, and satisfaction with work.
Controlled experiment on co-pilot effectiveness
An experiment with 95 professional developers was conducted to measure the speed gain from using co-pilot. Both a treatment group using co-pilot and a control group went through the task of writing an HTTP server in JavaScript. The treatment group achieved a 55% faster completion time and a higher success rate in fully completing the task compared to the control group. The experiment also revealed that less experienced programmers benefited more from co-pilot.
Best practices for trialing and adopting AI tools
When trialing and adopting AI tools like co-pilot, it is crucial to define what improvements the organization is seeking. Leaders should set clear expectations, allow sufficient time for adoption and training, and use developer surveys to assess the impacts. Patience and continuous evaluation through surveys at different intervals can help identify trends and gather feedback. It is also essential to involve early adopters as trainers and advocates within the organization to facilitate smooth rollout and adoption.
This week's guest is Eirini Kalliamvakou, a staff researcher at GitHub focused on AI and developer experience. Eirini sits at the forefront of research into GitHub Copilot. Abi and Eirini discuss recent research on how AI coding assistance impacts developer productivity. They talk about how leaders should build business cases for AI tools. They also preview what's to come with AI tools and implications for how developer productivity is measured.
Discussion points:
(1:49) Overview of GitHub’s research on AI
(2:59) The research study on Copilot
(4:48) Defining and measuring productivity for this study
(7:44) Exact measures and factors studied
(8:16) Key findings from the study
(9:45) How the study was conducted
(11:17) Most surprising findings for the researchers
(14:01) The motivation for conducting a follow-up study
(15:34) How the follow-up study was conducted
(18:42) Findings from the follow-up study
(21:13) Is AI just hype?
(26:34) How to begin advocating for AI tools
(34:44) How to translate data into dollars
(37:06) How to roll out AI tools to an organization
(38:47) The impact of AI on developer experience
(43:24) Implications of AI on how we measure productivity