Key Highlights

  • Corporate AI measurement is shifting from usage metrics like token consumption toward outcomes-based measures as budget scrutiny increases.
  • Usage metrics can't show real value, since AI-driven time savings often accrue quietly to individual employees rather than employers.
  • Google evaluates engineers on speed, ease, and quality together, to avoid rewarding faster output that compromises software reliability.
  • Research on Manufacturing firms found a "J-curve," where productivity dips before improving as employees learn new AI tools.

Usage Metrics Gave Way to Questions About Value

Early corporate enthusiasm for artificial intelligence often centered on maximizing usage regardless of cost, with organizations introducing internal leaderboards and tracking metrics such as employee token consumption. As usage rose, however, attention increasingly turned to whether that usage was translating into meaningful returns, a harder question to answer. Usage metrics retain some value for understanding adoption patterns. As one software company executive put it, engineering-heavy roles would be expected to consume more AI resources than others, but usage should not be zero anywhere in an organization if the technology is being meaningfully adopted.

The core limitation of usage metrics is that they say nothing about whether value is being created. Even when AI does generate genuine benefits, those benefits often accrue quietly to individual employees in the form of scattered time savings that are difficult for an organization to capture or measure, according to a researcher at LUT University in Finland. Time freed up for an employee does not automatically translate into a measurable benefit for the employer unless that time is redirected toward higher-value work.

Outcomes-Based Measures Must Account for Quality, Not Just Output

Shifting the focus to returns on AI Investment requires measuring outcomes such as team productivity or customer satisfaction, but these measures require careful design to avoid rewarding the wrong behavior. Academic researchers have found that AI has enabled scientists to produce a much larger Volume of published research, but not always research of comparable quality: a study examining papers derived from a particular health and nutrition dataset found publication rates rising from an average of four papers a year between 2014 and 2021 to 190 papers in the first nine months of 2024 alone.

To avoid this trap, some organizations measure AI's effect across multiple dimensions simultaneously. At Google, engineers have long been evaluated across three areas, speed, meaning how quickly a task such as a code review is completed, ease, meaning how much friction exists in processes like onboarding new developers, and critically, quality, to ensure that faster output does not come at the cost of reliable software, according to a Google Cloud executive.

Benefits Often Take Time to Materialize

Measuring AI's return is further complicated by the fact that benefits frequently take time to appear. Employees must first spend time learning to use new tools effectively, and increased output at one stage of a process can create bottlenecks elsewhere. Research examining AI adoption among American manufacturing firms identified a "J-curve" pattern in which productivity initially dips before eventually improving, an effect the researchers found was particularly pronounced at established firms and partly attributable to disruption in management practices that had previously been used to track performance. Any calculation of projected AI returns needs to account for this initial period of disruption rather than assuming benefits appear immediately.

Defining the Type of Return Matters

Organizations must also decide what kind of financial return they are seeking to demonstrate from AI investment. One straightforward approach is to use AI-driven time savings as justification for reducing headcount, though this can damage employee morale and eliminates the option of redeploying freed-up staff toward more productive work. A more measured approach involves calculating savings from avoided headcount growth rather than actual reductions, while attributing incremental Revenue directly to AI adoption is considerably more difficult and typically requires reliable baseline data and a culture of structured testing to isolate the technology's specific effect.

Given the ongoing uncertainty about how AI capabilities will continue to evolve, some organizations are adopting a third category of measurement focused on how well the transition itself is being managed, including employee satisfaction with AI implementation and progress in building internal expertise in areas AI is unlikely to automate.

Conclusion

The shift from usage-based to outcomes-based AI measurement reflects a broader maturation in how organizations think about the technology's value, moving past the assumption that more usage automatically produces more benefit. Whether this more disciplined measurement approach ultimately validates current levels of AI investment, or instead reveals a gap between spending and returns, will likely become clearer as more companies move past the initial adoption phase and into the multi-year period where J-curve effects are expected to resolve.