The vendor's slide says 10x. The pilot's internal report says the tool changed everything. The finance team, looking for the change in any number it actually tracks, finds nothing it can point to. Somewhere between the hype and the skepticism is the truth, and getting to it requires measuring AI's effect the way you would measure any real intervention, not the way you would sell one.
The good news is that credible measurements exist. The bad news is that they are more modest, and far more uneven, than the marketing implies, and that the easiest numbers to collect are the ones most likely to mislead.
What the honest studies find
Three field studies anchor the honest picture. In a randomized trial of 758 BCG consultants, AI raised throughput 12.2% and speed 25.1% on suitable tasks, with quality about 40% higher. In a study of more than 5,000 support agents, generative AI lifted resolutions per hour about 15% on average. In controlled experiments with software developers, GitHub Copilot cut the time to finish a coding task by roughly 55.8%. These are real, sometimes large effects, measured properly.
But notice two things. The numbers vary enormously by task and by tool, from 15% to 55%, which means any single company-wide figure is almost certainly wrong for most of the work it claims to describe. And the gains are wildly uneven across people: the support study found up to 34% for novices and near zero for the most experienced. An average hides more than it reveals. Measuring AI's impact with one blended number is like measuring a hospital by its average temperature.
The vanity-metric trap
The temptation is to measure what is easy: messages sent, documents drafted, prompts run, tokens consumed. These are activity metrics, and they are treacherous. The SPACE framework for developer productivity was written partly as a warning against exactly this, arguing that productivity is multidimensional and cannot be captured by any single count such as lines of code, and that activity metrics in isolation create perverse incentives. An AI tool can send an impressive number of messages and produce a mountain of drafts while moving nothing that matters. Counting the outputs of the tool measures adoption of the tool, not value from it.
The same confusion appears at the top line. Gallup's 2025 survey found that 65% of employees feel AI has helped their productivity, while only 12% strongly agree it has changed how work is actually done. Perceived help is real and worth knowing, but it is a sentiment, not an outcome. If measurement stops at how people feel, it will overstate the change; if it ignores how people feel, it will miss the leading signal. Both are data; neither is the result.
Measure whether the pump shipped
As Ricardo Semler has told it, after he collapsed at twenty-five he stopped counting whether people had worked the hours and started asking a different question: had the pump actually been shipped? He replaced the activity proxy with the outcome, and absenteeism fell once people were measured on what they delivered rather than how long they sat.
That is the discipline AI measurement needs. Pick the outcome that the work exists to produce, cases resolved, tickets closed, deals advanced, errors avoided, cycle time on a real deliverable, and measure that, before and after, ideally against a comparison group that did not get the tool. Then expect the answer to be uneven, and read the distribution, not just the mean. Where did the gains concentrate? Which tasks moved and which didn't? Which people? That texture is where the real management decisions live.
It is worth measuring the costs alongside the gains, too. A tool that speeds drafting but seeds subtle errors can look like a win on throughput and a loss on rework, and only a measure that includes quality and correction will catch the difference. The BCG trial is the reminder here: the same tool that lifted quality on suitable tasks degraded it on unsuitable ones. An honest scorecard tracks both edges of that blade, not just the flattering one.
Modest, uneven, and real
The mature position on AI productivity is neither the vendor's 10x nor the skeptic's shrug. The gains are real, sometimes large, highly task-specific, and concentrated among the less experienced. They are measurable if you track outcomes instead of activity, expect variation instead of a headline, and treat sentiment as a signal rather than a verdict. The organizations that will actually capture the value are the ones honest enough to measure whether the pump shipped, and patient enough to read where it did and where it didn't.
Sources
- Dell'Acqua, McFowland, Mollick, Lifshitz-Assaf, Kellogg, Lakhani et al. (2025). Navigating the Jagged Technological Frontier (Organization Science). RCT with 758 BCG consultants: on tasks inside AI's frontier, GPT-4 users did 12.2% more tasks, 25.1% faster, ~40% higher quality; on tasks outside it they erred more. https://pubsonline.informs.org/doi/10.1287/orsc.2025.21838
- Brynjolfsson, Li & Raymond (2025). Generative AI at Work (Quarterly Journal of Economics). Field study of 5,172 support agents: AI raised resolutions per hour ~15% on average, up to ~34% for novices and near zero for the most experienced. https://academic.oup.com/qje/article/140/2/889/7990658
- Peng, Kalliamvakou, Cihon & Demirer (2023). The Impact of AI on Developer Productivity: Evidence from GitHub Copilot (arXiv 2302.06590). Developers with Copilot finished a coding task ~55.8% faster than the control group; later RCTs replicate larger gains for less-experienced developers. https://arxiv.org/abs/2302.06590
- Forsgren, Storey, Maddila, Zimmermann, Houck & Butler (2021). The SPACE of Developer Productivity (ACM Queue 19(1)). Productivity is multidimensional and cannot be captured by a single metric such as lines of code; activity metrics in isolation create perverse incentives. https://www.microsoft.com/en-us/research/publication/the-space-of-developer-productivity-theres-more-to-it-than-you-think/
- Gallup (2025). State of the Global Workplace 2025. 65% of employees report a positive productivity effect from AI, but only 12% strongly agree AI has changed how work is done in their organization. https://www.gallup.com/workplace/659610/state-global-workplace.aspx