In late June 2021, Microsoft, owner of the collaborative web platform GitHub, and Open AI launched a technical preview of GitHub Copilot, an AI-based completion tool designed to help developers code faster. It’s actually been available to them since June 21 as a subscription, but is free for students and maintainers of popular open source projects. GitHub surveyed more than 2,000 developers who participated in the technical preview, the recently published study found that Copilot increases their productivity.
The launch of Copilot’s technical preview had sparked a lot of reactions in the developer community and, in particular, from the Free Software Foundation regarding copyright, ownership issues of AI-generated code and legal impacts for GitHub authors. It had then launched a call for white papers on co-piloting, copyright, machine learning and free software, selected and published 5 of them on February 24.
GitHub’s study on the productivity of developers using Copilot
Last April, GitHub had conducted a 1st study with 24 users to understand how programmers use and perceive Copilot, it had revealed that, although it did not reduce task time or increase success rate, most participants preferred to use Copilot in daily programming tasks, as it often provided a useful starting point and saved the effort of searching online. However, participants had encountered difficulties in understanding, editing, and debugging the code snippets generated by Copilot, which significantly hindered their task-solving efficiency.
For this new study, the research and engineering teams partnered to combine qualitative survey data from more than 2,000 U.S.-based developers with anonymized data to determine whether developers believe they are more productive using GitHub Copilot and whether the data proves they actually are.
GitHub Copilot usage correlates with improved developer productivity
Developers who report the highest productivity gains with GitHub Copilot are those who accepted the most code suggestions posted :
- Those who accepted 29.73% of the suggestions reported “huge” productivity gains;
- Those reporting “high” productivity gains had accepted 27.77% of the suggestions;
- Users who reported “medium” productivity gains had accepted 26.94% of the suggestions;
- 23.16% of suggestions were accepted by those reporting “modest” gains.