[ad_1]
Software engineer workflows have been transformed in recent years by an influx of AI coding tools like Cursor and GitHub Copilot, which promise to enhance productivity by automatically writing lines of code, fixing bugs, and testing changes. The tools are powered by AI models from OpenAI, Google DeepMind, Anthropic, and xAI that have rapidly increased their performance on a range of software engineering tests in recent years.
However, a new study published Thursday by the non-profit AI research group METR calls into question the extent to which today’s AI coding tools enhance productivity for experienced developers.
METR conducted a randomized controlled trial for this study by recruiting 16 experienced open source developers and having them complete 246 real tasks on large code repositories they regularly contribute to. The researchers randomly assigned roughly half of those tasks as “AI-allowed,” giving developers permission to use state-of-the-art AI coding tools such as Cursor Pro, while the other half of tasks forbade the use of AI tools.
Before completing their assigned tasks, the developers forecasted that using AI coding tools would reduce their completion time by 24%. That wasn’t the case.
“Surprisingly, we find that allowing AI actually increases completion time by 19% — developers are slower when using AI tooling,” the researchers said.
Notably, only 56% of the developers in the study had experience using Cursor, the main AI tool offered in the study. While nearly all the developers (94%) had experience using some web-based LLMs in their coding workflows, this study was the first time some used Cursor specifically. The researchers note that developers were trained on using Cursor in preparation for the study.
Nevertheless, METR’s findings raise questions about the supposed universal productivity gains promised by AI coding tools in 2025. Based on the study, developers shouldn’t assume that AI coding tools — specifically what’s come to be known as “vibe coders” — will immediately speed up their workflows.
METR researchers point to a few potential reasons why AI slowed down developers rather than speeding them up: Developers spend much more time prompting AI and waiting for it to respond when using vibe coders rather than actually coding. AI also tends to struggle in large, complex code bases, which this test used.
The study’s authors are careful not to draw any strong conclusions from these findings, explicitly noting they don’t believe AI systems currently fail to speed up many or most software developers. Other large-scale studies have shown that AI coding tools do speed up software engineer workflows.
The authors also note that AI progress has been substantial in recent years and that they wouldn’t expect the same results even three months from now. METR has also found that AI coding tools have significantly improved their ability to complete complex, long-horizon tasks in recent years.
However, the research offers yet another reason to be skeptical of the promised gains of AI coding tools. Other studies have shown that today’s AI coding tools can introduce mistakes and, in some cases, security vulnerabilities.
[ad_2]
Source link
