Red Hat’s October 28 announcement that it is joining a Berkeley-led AI efficiency project puts a practical question in front of enterprise buyers: what, exactly, should count as a saving when software agents do the work? A cheaper model call is useful. A cheaper completed assignment is the outcome that matters.
Red Hat Research says it and IBM Research will collaborate with the University of California, Berkeley’s Auto-Optimized Agents team in the STRIDE AI Efficiency Challenge. The project aims to use records of real usage to tune an open-source system for deployed coding and enterprise agents. The announcement describes a research effort, not a demonstrated reduction in customers’ operating costs.
Where the measurement boundary falls
The wider program makes that distinction consequential. In its October 14 announcement, the National Science Foundation described eight teams developing and deploying efficiency technologies and a ninth team developing measurement standards. Start2 Group’s STRIDE announcement identifies MLCommons as that benchmarking team. It also lists approaches spanning data processing, infrastructure and multi-turn agents.
TENS Magazine’s analysis is that these projects need a common definition of completed work before their improvements can be compared. Hardware utilization, response speed and task completion answer different questions. An improvement in one can be valuable without establishing an improvement in the others. A buyer needs to know which claim an experiment actually supports.
Consider a hypothetical coding assignment with a fixed acceptance test. One system may produce a patch quickly, then require several attempts to fix it. Another may spend longer checking its first proposal and finish successfully. Measuring only the first response would favor the wrong system if the buyer’s priority were the total time to an accepted patch. This is an illustrative scenario, not a reported result from the challenge.
Count the work that did not finish
TENS proposes a simple accounting rule: divide the cost of all attempts in a test batch by the number of assignments that meet a predeclared acceptance standard. Keep unsuccessful assignments in the cost numerator. Report their number separately so that abandoning difficult work cannot quietly improve the headline figure.
For illustration, suppose a baseline spends 100 accounting units and completes 80 assignments. Its cost per completion is 1.25 units. A revised system spends 80 units but completes only 60. Its cost per completion is about 1.33 units: total spending falls 20 percent, while spending per successful assignment rises roughly 6.7 percent. These invented inputs demonstrate the arithmetic; they are not performance claims about Berkeley, Red Hat or any participating team.
That ratio is still insufficient on its own. TENS would publish it alongside the completion rate, elapsed time and the distribution of task difficulty. Otherwise, a system could appear economical because it handles easy requests well while failing the assignments that motivated its purchase. Human review time should be recorded separately, with the same review rules applied to both systems.
Real usage needs a comparison that stays fair
The Berkeley project’s use of operational traces creates an opportunity to study work as it actually unfolds. It also raises a question for evaluation: how should an organization distinguish a better system from a more favorable mix of requests? The public project descriptions do not settle that question or supply a completed comparison.
TENS proposes keeping two evaluation sets. The first would hold tasks and acceptance rules constant so successive versions can be compared. The second would contain later, previously unused work to reveal whether the improvement survives changes in demand. A report should disclose both results rather than folding them into one average.
The first set tests whether a change helped on a stable workload. The second tests whether that benefit travels. A gap between them would be useful information, not automatically a failure: it could identify where an optimization needs retuning. But the distinction should be visible before a limited result becomes a broad promise.
Separate deployment milestones from savings
NSF says teams are expected to reach deployment at scale or benchmarking standards within two years, with a faster route for teams reaching deployment within one year. Those are program objectives. They do not establish how much any particular agent will save, and deploying software is not itself evidence that it improves the cost of accepted work.
For buyers following this effort, TENS would look for a compact record connecting each claimed gain to a named workload, a fixed success rule, all attempted work and the resources counted. Energy, computing charges and reviewer time should remain distinguishable. Combining them requires explicit assumptions; a lower cloud bill alone cannot demonstrate lower energy consumption.
The significance of the challenge lies in its attempt to connect optimization with operational deployment and measurement. Its eventual evidence will be most useful if teams expose where the savings stop. For an enterprise agent, that boundary should extend through the retries and checks to the work a user can actually accept.
Image: Archival Columbia supercomputer at NASA’s Advanced Supercomputing Facility; illustrative computing infrastructure, not the STRIDE project. Credit: Trower, NASA via Wikimedia Commons. License: Public domain (PD-USGov-NASA). Modifications: center-cropped and resized from 2100 × 1524 to 1200 × 675 pixels; no generative edits.
