Measuring the real impact of AI on developers’ work requires distinguishing between feelings and verifiable data. Code generation tools are multiplying, and the promises of productivity gains are too. The DORA 2025 report and several recent controlled trials allow us to confront these promises with the results observed in the field.
AI Productivity and Software Stability: What the DORA 2025 Report Measures
The DORA 2025 report establishes a positive link between AI adoption and delivery throughput. Teams using code assistants deliver more features over a given period.
This same report identifies a persistent negative link with the stability of production deployments. In other words, producing faster does not guarantee producing better. Post-deployment incidents do not decrease proportionally to the code delivered.
A point often overlooked in competing analyses: AI tends to amplify existing practices within a team. A team with good review and testing practices sees its results improve. A team with fragile processes sees its weaknesses worsen. The tool is not corrective; it is amplifying. Resources like codyx.org allow for exploring different code generation approaches in this context.
Controlled Trials on Code Generation: Contradictory Results
The available studies do not converge on a single conclusion. The table below summarizes the discrepancies observed between two types of recent trials.
| Type of Trial | Context | Observed Result |
|---|---|---|
| METR 2025 Trial | Experienced developers, familiar open source projects | Measured task duration increased despite a feeling of acceleration |
| Field Trials (3 companies) | Common development tasks, access to Copilot | More tasks completed during the observed period |
The METR trial highlights a striking gap: developers felt that AI had accelerated them, while the actual duration had increased. This perceptual bias poses a methodological problem for any company evaluating AI solely based on its teams’ feelings.
The field trials, on the other hand, show a measurable gain in the volume of completed tasks. These results do not measure the same thing and cannot be generalized to all contexts. The type of project, familiarity with the codebase, and the developer’s level of experience radically change the equation.
Security of AI-Generated Code: An Underestimated Angle
Most productivity analyses focus on writing speed. The security of the produced code remains a poorly addressed operational angle.
Security verification and human review can absorb part of the time gained in the initial writing. A developer who generates a function in a few seconds but then spends twenty minutes checking that it contains no injection flaws or mismanagement of permissions has not necessarily gained net time.
This phenomenon creates a shift in workload rather than a reduction. Writing time decreases, review time increases. For teams working on sensitive applications (health, finance, infrastructure), this transfer deserves quantification before proclaiming overall gains.
- Automatic generation does not verify compliance with project-specific security standards
- Unit tests generated by AI often cover nominal cases but neglect edge cases and attack scenarios
- Human code review remains the last line of defense, and its time increases as the volume of produced code grows
Measuring Security Separately from Speed
Evaluating a code generation tool solely by the number of lines produced or tasks closed is akin to measuring a vehicle’s speed without checking its brakes. Security metrics (vulnerabilities detected post-generation, correction rates before deployment) should be included in any AI adoption dashboard.
Developer Sentiment vs. Measured Data: How to Evaluate AI
The gap identified by the METR trial between perception and measurement raises a practical question for technical leaders. On what indicators should a decision to adopt or renew a license be based?
- The volume of tasks completed per sprint gives a signal about throughput but not about the quality of the delivered code
- The rollback rate after deployment (DORA metric) captures actual stability better than team sentiment
- The code review time per pull request, measured before and after AI adoption, reveals whether the writing gain turns into a verification overload
A reliable evaluation combines at least three indicators: throughput, stability, and review time. Relying solely on sentiment or the volume of code produced exposes one to erroneous conclusions.
The DORA 2025 report shows that the teams that are truly progressing are those that already had solid practices in automated testing and systematic review. AI has not replaced these practices; it has made them more productive. For others, the tool has primarily accelerated the production of technical debt.
The challenge for the coming months is not whether AI generates code faster. It does. The open question concerns the real cost of this code once deployed, maintained, and secured over time.
