THE PATTERN
● NVIDIA Vera CPU: 88 cores, chosen for single-thread speed on orchestration rather than core count.
● NVIDIA Groq 3 LPX: a separate chip purely for latency-critical decode.
● OpenAI Jalapeno: full-stack design minimising data movement, keeping model state local.
● The shared assumption: the workload is now an agent iterating, not a batch job completing.
What changed in the Vera CPU
At Hot Chips on 24 August, NVIDIA detailed the Vera CPU: 88 custom Olympus cores split across six chiplets on a single interposer, with LPDDR5X memory and NVLink-C2C for pairing with a GPU or a second CPU.
The interesting part is not the core count. It is that NVIDIA describes the design as prioritising single-thread performance for orchestration and tool-calling instead of raw core count — claiming roughly 1.8x speedup on agentic workloads and up to 30x throughput against Grace Blackwell in specific interactivity scenarios.
WHY SINGLE-THREAD, WHY NOW
An agent loop is not parallel work. It decides what to do, calls a tool, waits, reads the result, decides again. That chain is serial by nature, and adding cores does not speed up a sequence of dependent steps.
For a decade, server CPU design has meant more cores. Optimising for the opposite is a deliberate bet that orchestration, not parallelism, is the bottleneck.
The same bet, three different ways
| Design |
What it optimises |
What it gives up |
| Vera CPU |
Single-thread speed for orchestration |
Raw parallel core count |
| Groq 3 LPX |
Token generation latency |
Generality — it only does decode |
| OpenAI Jalapeno |
Work per watt, keeping state local |
Training entirely — inference only |
Three organisations, three architectures, one assumption: the thing being optimised is an agent working through steps, not a model answering a question.
Why this follows from the software data
The hardware is catching up to something the telemetry has been saying all month.
Linear published data from its paid workspaces showing teams using coding agents went from 21 weekly pull requests to 65, while total development time rose — because review scales with volume. Salesforce measured enterprise agent deployments going from five per organisation to thirteen, with build time down 53 percent. Cloudflare shipped a browser and a payment protocol built for agents rather than people.
Agents generate tokens across hundreds or thousands of steps. Each step waits on the previous one. That workload profile did not exist at scale two years ago, and it does not resemble the batch inference these platforms were designed around.
What it means for you
| If you are... |
The useful read |
| Choosing an inference provider |
Ask about time-to-first-token and inter-token latency, not just throughput. Those are now different products |
| Building agent workflows |
Latency compounds across steps. A 200ms difference per step is minutes across a long session |
| Reading benchmark claims |
Check whether the metric is tokens per second in aggregate or per user. They diverge sharply |
| Just using an API |
You will feel this as agents becoming less frustrating, over quarters rather than weeks |
The honest caveat
Every figure here comes from the company selling the hardware. NVIDIA's 1.8x and 30x are NVIDIA's numbers on NVIDIA's chosen scenarios. OpenAI's 1.5 to 1.9x is OpenAI's own testing. Independent benchmarking of any of it is months away.
What is not a claim is the design direction. Three organisations independently deciding to optimise for serial agent steps rather than parallel throughput is a fact about where they think the demand is going, whatever the multipliers turn out to be.
FAQ
What is different about the Vera CPU?
88 custom Olympus cores across six chiplets, designed to prioritise single-thread performance for orchestration and tool-calling rather than maximising core count. NVIDIA claims roughly 1.8x on agentic workloads and up to 30x throughput against Grace Blackwell in specific interactivity scenarios.
Why does single-thread performance matter for agents?
An agent loop is serial — decide, call a tool, wait, read, decide again. Each step depends on the previous one, so adding parallel cores does not speed up the chain.
Is this a real shift or marketing?
The multipliers are vendor claims. The design direction is real: NVIDIA split decode onto a separate accelerator and redesigned a CPU around orchestration, and OpenAI built a chip that keeps model state local. Those are expensive decisions, not slogans.
Does this make current hardware obsolete?
No. Training is unaffected and batch inference still favours throughput. This is a new class of optimisation alongside the existing one, not a replacement.
What should I measure instead of throughput?
Time to first token and inter-token latency, per user rather than in aggregate. For agent workloads those determine whether a session feels responsive, and they can be poor on a platform with excellent total throughput.