GitHub CoPilot Wont Replace (Many) Software Engineers

September 14, 2022

Shakespeare as a robot in a Science fiction movie style via Dalle 2

GitHub Copilot is a code completion AI based on GPT3 that integrates with popular IDEs (e.g. VSCode) that fully launched in June 2022. I’m rebuilding some of my old PHP projects in JavaScript & Python, and decided to give it a go.

To get setup:

  1. Signup with your GitHub account
  2. It costs $10 p/m, with a free 2 month trial, and free for students and open source maintainers
  3. Install the CoPilot extension in your IDE e.g. VS Code

To use CoPilot you give it a prompt as a comment or a line of code e.g. in JavaScript:

// Check if local storage is empty, and if not, populate the form with the values

And you keep pressing tab until it stops suggesting lines of code e.g. comment in green, suggestion in grey: Screenshot showing CoPilot code completion

I was blown-away at how well CoPilot performed for the small project I was working on, but it’s limitations became clear over time.

What it does well

  • It is spookily efficient at suggesting code for common tasks
  • Its suggestions were in the context of my existing work
  • It a great job working within a single file

What it doesn’t do well

  • It didn’t always do a good job of understanding the contents of other files in the project
  • For whatever reason it really struggled with CSS (which I could really use the help with)
  • It doesn’t plan your project for you, it just gives you the code you ask for

CoPilot will reduce the demand for small projects e.g. rather than using Fiverr to create a scraper or a simple form, people with a little technical skill will build their own. For example I created this simple Python script by typing a couple of comments and pressing tab a lot:

Screenshot showing CoPilot generated Python script

However ultimately CoPilot suffers from the same limitation as GPT3; it’s an incredible guessing machine, but it has no context. Maybe in future it can work in tandem with ‘bigger picture’ models, but without context, CoPilot can’t plan ahead in the same way as a human.

The big loser from this kind of technology might turn out to be Stack Overflow. Normally I would check it, via a Google search, anytime I get stuck, but as CoPilot was suggesting syntax for me there was no need.

I’ll pay the $10 p/m - I can forsee that it will be a big productivity win for me and others who work on small projects, automate tasks etc, but it won’t be replacing professional software engineers.

Header image courtesy of Dalle-2.

Chris Reynolds is a Bay Area Product Manager with 15 years of international experience in SEO, digital marketing, UX, analytics and team management.

© 2022 Chris Reynolds