348.41 Wh. That is what one Reflexion agent on Llama-3.1-Instruct 70B spent answering a single HotpotQA question, 136.5 times the energy of the same model doing one-shot inference.
The number comes from a KAIST group led by Minsoo Rhu, whose paper on the cost of dynamic reasoning was accepted at HPCA-32. They instrumented the hardware rather than reading a vendor bill. Reflexion costs 2.2 times what LATS costs per query on identical hardware and the same benchmark. Two frameworks a team might pick between on a Tuesday afternoon based on an accuracy table differ by more than a factor of two on operating cost.
That spread is not the most actionable finding. This is: GPUs sit idle for as much as 54.5% of agent execution time. The agent loop is a sequence of blocking I/O operations — a search call, a retrieval index, a sandboxed code run — interleaved with bursts of generation. If you have provisioned dedicated accelerators per agent worker, you are paying full rate for roughly half a machine.
If you have provisioned dedicated accelerators per agent worker, you are paying full rate for roughly half a machine.
This is why the compute conversation is moving. The bottleneck in agentic AI is not the model generating tokens. It is everything happening around the model.
The Pragmatic Engineer's Pulse newsletter describes the trend plainly: after a GPU shortage and a memory shortage driven by AI companies, we are now experiencing a CPU shortage, because AI agents use a lot more CPU with tool usage. The advice is blunt: if you will need more compute in the future, secure it now.
The CPU was never supposed to matter this much. In the chatbot era, a CPU acted as the head node for a server with four to eight GPUs — handling scheduling, I/O, and system management while the GPUs did the heavy math. AI data centers operated at CPU-to-GPU ratios of roughly 1:4 to 1:8.
Then agents arrived. Instead of answering one prompt, an agent breaks a goal into steps, decides what to do next, calls multiple models, queries databases, connects with APIs, runs enterprise applications, checks permissions, retrieves memory, validates output, and loops back through the process again. AMD's infrastructure team lists what falls on the CPU: orchestration, agent execution and tool calls, policy and security checks on every autonomous action.
The ratio is shifting toward 1:1, and in some enterprise workflow automation cases, higher on the CPU side. Arm estimates that while traditional AI data centers require about 30 million CPU cores per gigawatt, demand will surge to 120 million CPU cores per GW in the agent era — a fourfold increase.
The latency data backs this up. The CPU-centric characterization paper from Georgia Tech and Intel found that tool processing on CPUs can account for up to 90.6% of total latency. CPU dynamic energy consumption can reach 44% of the total dynamic energy at large batch sizes.
Ben Bajarin's model at The Diligence Stack puts the GPU idle window from agent orchestration at roughly 12 to 22 percent of total inference time, scaling with agent complexity. A CPU rack costs roughly $300K fully loaded. A GPU rack costs roughly $4 million and draws about seven times the power. At a 5 percent CPU power allocation in a 1GW facility, Bajarin's model shows a 2 percent increase in effective token throughput and a 3.7 percent decrease in cost per token, with only a 1.7 percent increase in total capex. The breakeven sits at roughly 10 percent allocation.
The market has already responded. In March 2026, Nvidia began selling its Vera CPU as a standalone product. Arm unveiled its first in-house CPU, the Arm AGI CPU, ending 35 years of pure licensing. That a GPU company and an IP licensing firm both entered the CPU market in the same month is not a coincidence. Intel and AMD raised prices across select CPU product lines toward the end of the first quarter.
The scheduling problem is harder than "add more CPUs." A Brown University paper profiled 19 AI tools across GPU and CPU and found that 11 were GPU-preferred, 4 ambiguous, 1 CPU-preferred due to PCIe transfer dominance, and 3 device-neutral. Blanket GPU-first scheduling, the default in LangGraph and LangChain, is suboptimal.
Microsoft is rebuilding Windows around this reality. The next version ships with native agent identification, a centralized registry of locally available MCP servers, and an OS-agnostic isolation mechanism for agents that run tools safely. WindowsML is a new hardware abstraction layer for running models across GPU, NPU, and CPU. The OS aims to ship Small Language Models without requiring an NPU at all.
Nick Williams makes a point that cuts against the entire framing of AI's energy footprint. The conversation, he argues, is almost entirely focused on inputs — training runs, inference queries, GPU hours. It overlooks a different category of waste: the energy generated by the data architecture decisions of the organizations that rely on AI. That waste does not show up in any model card. It shows up in the bill.
The practical implication for teams building agents is not to abandon GPUs. A better-provisioned CPU orchestration layer improves effective GPU utilization. The CPU does not compete with the GPU. It raises the return on the GPU asset by reducing idle time.
But the ratio assumptions embedded in most infrastructure plans are training-era assumptions. They were built for a world where the dominant job was training large models or serving turn-based inference. That world is ending.
The teams that internalize this first will provision differently. They will separate the CPU tier from the GPU tier instead of treating the CPU as a head node. They will measure GPU idle time as a first-class metric, not just token throughput. They will pick agent frameworks with the energy cost in the room, not just the accuracy table.
And they will secure compute now, because the shortage is already here.