Beyond Tokens: Why Intelligence Per Watt Is The Metric That Matters
By Dr. Steven Woo, fellow and distinguished inventor at Rambus.
gettyAI progress centers on tokens: tokens processed, tokens generated, tokens per second and tokens per watt. And because tokens are easy to count, they’ve become the default unit of AI performance. But token-based metrics are only part of the story and miss what users really want. A more useful metric, and one closer to the industry’s goal, is how much intelligence a system delivers per watt.
“Intelligence Per Watt: Measuring Intelligence Efficiency Of Local AI,” published in November 2025 by a Stanford team, found that small local models can accurately answer 88.7% of single-turn chat and reasoning queries. Intelligence efficiency—capability delivered per watt—improved 5.3 times between 2023 and 2025, roughly 3.1 times from better models and 1.7 times from better hardware—factors that compound rather than add. This efficiency, not pure token throughput, will determine how well AI can keep scaling.
Ask a CFO, CIO or CTO running AI workloads what worries them, and it’s rarely whether they’re generating enough tokens. Although tokenmaxxing has become a metric that some companies use to measure AI engagement, others see the dangers of simply measuring token consumption independent of productivity and delivered value. Several companies are cutting back on token spending, with Uber among others stating in May 2026 that it had already spent its 2026 AI budget, and their COO adding it was becoming “harder to justify” their internal AI costs. There’s a growing call for valuemaxxing, which directs teams toward measurable outcomes based on the value being generated by these tokens.
Concerns about measuring token usage are compounded by the fact that reasoning models don’t produce a single response per query; they work through an internal chain of intermediate tokens the user never sees. Published work on model development puts these generations in the thousands to tens of thousands of tokens. That means a query can consume far more resources than its visible output suggests.
The pressure to continuously improve value and reduce the cost per token extends beyond inference bills. TrendForce expected server DRAM contract prices to rise 13% to 18% quarter over quarter in Q3 2026, so memory, one of the critical workhorses for AI processing, is getting more expensive as it’s being asked to do more.
For hardware companies, that changes the questions being asked. It stops being how many tokens a chip can generate and becomes how much bandwidth memory can deliver per dollar and per watt, and how these metrics will improve in the future.
For decades, the semiconductor industry relied on Dennard scaling to improve power efficiency. As transistors shrank, they drew less power, so more compute arrived without a matching jump in energy use. That relationship broke down in the mid-2000s as transistors continued to shrink, but the automatic power benefit that used to come with each new process node no longer followed.
With this lever gone, the industry had to substitute architectural changes for what used to be a by-product of process scaling. Memory is where much of that effort is going: Moving data is a major consumer of system power, making it one of the largest levers to pull.
The case for stacking DRAM directly on top of a processor has been argued for years. The energy needed to move a bit scales with distance. A DRAM stack beside a processor sits 10 to 12 millimeters away. But stacking the DRAM on top of a processor cuts the distance to about a millimeter. Bandwidth improves separately: Memory beside the processor feeds data through the die’s edges, so interconnect scales with perimeter; stacking can move data through the whole face of a die, allowing bandwidth to scale with area.
Stacking DRAM has become a focus topic of the semiconductor industry. At Hot Chips 2026, Samsung released more details about zHBM, which places the processor beneath the DRAM stack and claims roughly 70% lower I/O power than the previous generation. That’s what more bandwidth per dollar and per watt looks like.
Stacking has manufacturing and engineering challenges. Although silicon conducts heat reasonably well, a stack of DRAM sitting on a hot processor acts like a blanket and prevents heat from being moved away. Samsung’s answer is thinner dies and smaller four-high stacks rather than the 12-high or 16-high configurations conventional HBM supports. The energy saved on data movement is coupled with less thermal resistance to achieve a high performance, thermally viable architecture.
The same constraint is reshaping infrastructure above the die. Enterprise racks that once drew 20 to 30 kilowatts are heading toward hundreds of kilowatts, and eventually toward a megawatt. Power Electronics Magazine reported in January 2026 that 54 volt in-rack distribution “isn’t designed to support the megawatt-scale racks,” and that the transition to 800 volt DC begins in 2027. Copper is expensive and limits current, so when power demand climbs, voltage is the remaining variable that can rise.
Delivering the most intelligence for a given dollar and a given watt used to be determined by two scaling trends running together: Moore’s law delivered more transistors per dollar, and Dennard scaling delivered them at lower power. Moore’s law continues, albeit at a slower pace. Dennard scaling doesn’t, so the power portion of that result now has to be engineered deliberately, architecture by architecture.
The industry has hit walls like this before, and the solution was rarely a single breakthrough. It was a return to fundamentals: measuring the right thing and re-architecting around the real constraint instead of the metric that’s easiest to report.
For the CFOs, CIOs and CTOs approving AI budgets, and for the hardware and software teams underneath them, the number worth tracking isn’t simply how many tokens a system produces. It’s how much usable intelligence and value these tokens return for the dollars and the watts spent producing them.
Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?
