Timing Artificial Tasks A widely cited study found that developers who used GitHub Copilot completed a task 55% faster than those who did not [Peng2023]. The task was implementing an HTTP server in JavaScript from scratch, in ninety minutes; the developers had no other obligations that day. Real software development involves navigating a large codebase you did not write, understanding a requirement described ambiguously in a ticket, coordinating with colleagues, and attending meetings. Speed on a greenfield toy task does not predict speed on any of that. A randomized controlled trial with experienced open-source developers found the opposite of what participants themselves predicted: giving them access to AI tools increased task completion time by 19% [Becker2025].

Comparing AI to Nothing Studies that compare AI-assisted developers to a control group using nothing have chosen a baseline that does not exist in practice. Developers without LLM assistants use documentation, colleagues, and the time they would otherwise spend thinking through the problem themselves. The relevant question is whether LLM tools outperform the alternatives developers already have, and that comparison is rarely made [Peng2023]. Choosing a weak baseline makes any tool look good; it does not make the tool useful.