An automated pair programmer: fact or fiction?
Oege de Moor, Alex Gravely, Albert Ziegler – GitHub OCTO. August 31st, 2020
We evaluate the use of OpenAI’s language models trained on source code, for the specific task of Python code synthesis from natural language descriptions.
Our findings are as follows:
Out of 233 hand-crafted programming exercises supplied by 30 GitHub engineers, 93% are successfully solved. The exercises include StackOverflow-type problems involving the use of an unfamiliar package, as well as programming challenges typically used in coding interviews at high-tech companies, and some elementary examples. The success in solving self-contained programming problems demonstrates that an “automated StackOverflow’ is around the corner.
ON 58,391 functions taken from open source repositories, the model achieves a 52.3% success rate in creating an alternative implementation from the original documentation of that function, which passes the test. Furthermore, we project that with more computational power the success rate increases to 60%. There are obvious ways in which the success rate can be further improved, even with today’s OpenAI models, by providing the model with more context about the repository that contains the function. These results prove and IDE plugin that helps developers write non-trivial code is not far away.
While the OpenAI models are already amazing today, they are further improving at a ferocious pace — just a few weeks ago the success percentage on rewriting arbitrary functions was 43.3%, instead of 52.3%. Where that earlier model needed 150 attempts to find a correct solution, the newer model needs 14. We therefore anticipate applications way beyond mere program synthesis, where developers create new code in an interactive conversation with the model.
We conclude that OpenAI’s technology is poised to change developer tools in fundamental ways. In particular an ‘automated pair programmer’ can be built that puts the collective knowledge of the entire GitHub community at the fingertips of every individual.
An automated pair programmer, which puts the collective knowledge of the entire GitHub community at the fingertips of every individual, would transform the software industry. Many aspiring developers who do not have access to adequate training and advice today would instantaneously become productive…