Twenty-six days. That is how long it took the SGLang team to close a 100x performance gap on DeepSeek V4 running on AMD's MI355X, a chip that at launch could only run FP8 inference and little else.
In the old mental model of the CUDA moat, that number is impossible. In the new one, it is the whole story.
Start with the tell that the game already changed. In April 2026, OpenAI's enterprise agentic spending overtook its ChatGPT spending. The workload that pays the bills is no longer a chatbot answering one question at a time.
SemiAnalysis noticed, and spent more than $3 million building a new benchmark, AgentX, because the old way of measuring performance had quietly stopped describing production. Fixed-sequence-length tests—8k in, 1k out—measure a workload that no longer exists at scale.
Reality, in their words, is "multi-turn, long context, high prefill reuse, with sub agent bursts, KVCache offload, and numerous tool calls." When the shape of the workload changes, the shape of the moat changes with it. That is the argument.
Here is the mechanic that matters most. In an agentic session, the conversation grows linearly: the output of turn n-1 gets concatenated to the input of turn n. Context accumulates—system prompts, tool definitions, hundreds of turns.
Which means most of what a model sees on any given turn, it has already seen. As the number of turns grows, the ratio of cached input to uncached input tends toward 1.
Sit with that. Toward 1 means almost none of the compute is new. The frontier of the work is not raw matrix multiplication on fresh tokens—it is finding the cached prefix and reusing it.
SemiAnalysis states the conclusion bluntly: agentic inference is "inherently a systems problem." Because prefix reuse is so high, the KV tensors that hold that cached context must move efficiently across nodes and ranks—the domain of tools like NIXL, Mooncake, and MORI-IO.
And conversations must be routed to whichever node already holds the right prefix, to maximize cache-hit rate—the domain of LLM-d, Dynamo, and the vLLM and SGLang routers.
Notice what is absent from that list: the fastest single kernel. The decisive advantage has migrated up the stack, from per-kernel CUDA fluency to cache-aware systems engineering.
The economics make the stakes concrete. A simple web-agent task—extract a field, fill a form, navigate a site—is 10 to 20 inference calls. A multi-site workflow runs into the hundreds.
Yutori's Navigator, per its co-founder, ships web tasks at 2-5x cheaper and faster than closed models. The lever is not a magic chip. It is a roughly 80% prefix cache hit rate, plus speculative decoding on a smaller footprint.
When four out of five prefill tokens come from cache, whoever manages that cache best wins the unit economics. The chip underneath becomes a commodity input to a systems problem.
When four out of five prefill tokens come from cache, whoever manages that cache best wins the unit economics. The chip underneath becomes a commodity input to a systems problem.
Now the pivot, because this reframing is what makes the parity results legible instead of shocking.
When DeepSeek V4 dropped, NVIDIA's CUDA stack had Day-0 support on GB200 and GB300 through vLLM and SGLang—open-box, stable, expected. But Huawei's Ascend 950DT, running the CANN framework, also hit Day-0 full-stack support on the same day.
A Chinese silicon-and-software stack reaching functional parity with CUDA on release day is not supposed to happen if the moat is per-kernel software depth. It happens easily if the layer that matters is a shared, open serving stack.
And that stack is shared. AgentX has already generated over 70 upstream pull requests optimizing real agentic workloads across vLLM, SGLang, TensorRT-LLM, ATOM, AITER, Dynamo, LMCache, and Mooncake.
Read the contributor list and you see AMD's Shanghai team, NVIDIA's Shanghai TensorRT-LLM team, and Huawei-adjacent maintainers all improving the same open serving layer. Every optimization to prefix routing or KV movement improves it for every vendor at once.
That is why AMD's 100x-in-26-days is a systems result, not a silicon one. The MI355X did not change. The serving stack around it did, on an open codebase where the routing and cache logic is vendor-portable.
The switching cost CUDA actually protected was never just good documentation. It was that developers had to invest real time learning CUDA specifically, and AMD developers had no equivalent shortcut. That asymmetry is what is dissolving.
AMD is pressing directly on it. At Advancing AI 2026 it introduced ROCm.ai, folding a ROCm CLI, AMD Skills, and the open-source Hyperloom optimizer into coding agents like Claude, Cursor, and Codex—turning generic GPU advice into hardware-specific guidance inside the tools developers already use.
NVIDIA had comparable infrastructure first: a CUDA MCP Server, an Nsight AI Copilot, and ComputeEval, a benchmark built to grade LLM-written CUDA. But sequencing is not consequence.
Whoever wins the agent-tooling race, the race itself undercuts the specific advantage CUDA depended on. Once an agent can onboard a developer onto either platform, the fluency gap that anchored switching costs converges rather than tipping.
There is a culture layer here too. Anthropic's head of compute described using Claude over a weekend, with a "/goal" command, to bring up an internal inference stack on AMD hardware. When the moat was hand-tuned kernels, that weekend takes a specialist a quarter. Agentic tooling collapses it.
So the moat is being drained, not breached. Nobody wrote better rival software that beat CUDA on its own terms. The workload changed shape, and the new shape rewards a layer where CUDA was never the lock.
Which brings the necessary counterweight, because two asymmetries survive and one does not.
The one that survives is silicon and networking. NVIDIA's raw hardware lead is real, its data-center revenue set records last quarter, and the pie is growing so fast that AMD gaining share does not mean NVIDIA shrinks.
SemiAnalysis is explicit that AMD's rack-scale challenge, Helios, is going through production ramp hell—a slow build hampered by a weak SerDes design requiring over 550 Broadcom retimers per rack. Parity in the serving stack does not confer parity in packaging thousands of chips into a coherent system.
And CUDA's grip on training is not the thing agentic inference dissolves. Training is where hand-tuned performance, tight networking, and mature tooling still compound into a genuine advantage.
The asymmetry that does not survive is developer lock-in to CUDA specifically for inference. That was the switching cost the moat was built to protect, and it is the exact thing agentic workloads and agentic tooling erode from both directions—the workload no longer rewards kernel fluency, and the tooling no longer requires it.
The broader market is already voting. A Futurum survey put GPUs at 58% of data-center compute spending in 2025, but for 2026 expects XPUs—neither GPU nor CPU—to lead growth at 22%, ahead of GPUs at 19%. More than 50% of tokens on AWS Bedrock already run on Trainium.
The lesson is not that CUDA is finished. It is that a moat protects a specific piece of terrain, and the fighting has moved.
For the fastest-growing slice of inference, the decisive question is no longer who writes the fastest kernel. It is who moves and routes KV tensors best across a cluster—and that answer is being written in open source, for everyone, at once.