The Short Version
This is the rarest thing in model comparison: one benchmark, one physical task, one harness, four models. No vendor-reported figures, no partial credit, no arguing about task sets. The car either got round the course or it did not.
The Results
| Model |
Best score |
Detail |
| GPT-6 Astra |
100% |
Finished in 5:22 on the second attempt |
| Claude Fable 5.1 |
45% |
Reached 45% on the third attempt |
| Grok 4.6 |
11% |
Never progressed past the first corner |
| GPT-5.6 Sol |
6% |
Minimal progress across all attempts |
Three attempts each. Astra's two runs cost roughly $9.75 in tokens combined.
The Setup
A 2022 Toyota Corolla fitted with comma four hardware running openpilot, on a 134.7-metre course in a Bay Area parking lot: a left entry turn, a long straight aisle, gentle bends, two right turns into cross aisles, and a parking zone.
Models drove through three MCP tools - observe() for camera frames and metrics, set_motion() for steering and speed, stop_now() for emergency braking. Speed was capped at 3.5 m/s with multiple safety layers, and the successful run averaged under 1 mph.
Built by Aditya Ramabadran, Simon Mahns and Tobias Gessler.
Why the Failure Modes Matter More Than the Scores
Three failure patterns showed up, and only one of them is about intelligence.
Latency. Models spent too long thinking between commands. A car does not pause while you reason. This is the failure that a smarter model does not fix and a faster one does, and it is the single clearest argument that agentic control is a latency problem before it is a reasoning problem.
Spatial grounding. Models misjudged which side of a cone boundary to follow, and misunderstood the car's width relative to what the camera showed. Knowing a rule is not the same as knowing where your body is.
The first corner. Grok 4.6 never got past it. When a model fails at the first state transition rather than degrading gradually, the problem is usually the initial grounding, not the task.
What This Does Not Show
This is not autonomous driving and nobody involved claims it is. Under 1 mph, a parking lot, cones, a cap of 3.5 m/s and multiple safety layers. A purpose-built driving stack would do this better, faster and more cheaply, and does, every day.
What it shows is that a general-purpose language model can now hold perception, planning, control and latency together well enough to complete a physical task end to end. Six months ago that did not work at all.
Why This Benchmark Is Worth More Than Its Score
Every other comparison this week is vendor-reported. Anthropic and OpenAI publish computer-use figures on different harnesses with different task sets, and the gap between how two vendors score the same model exceeds most model-to-model gaps - we go through that in the computer-use comparison.
DrivingBench has none of that. One course, one car, one set of tools, pass or fail. It is a small result measured cleanly, which at the moment is rarer and more useful than a large result measured by the people selling the model.
Decision Framework
- Building anything real-time or embodied - Astra is the only model here that finished, and latency is why. Budget for it.
- Evaluating agentic control generally - measure time-to-command, not just answer quality. Three of four models failed on clock, not on reasoning.
- Choosing a model for a physical or robotics pipeline - do not choose from this. Use a purpose-built stack and keep the LLM for the parts that need language.
- Reading benchmark claims this week - prefer results where one party ran every model on one harness. This is one of very few.
Verdict
GPT-6 Astra is the only model that finished, and $9.75 of tokens to drive 134 metres at walking pace is both an achievement and a reminder of how early this is. The transferable finding is the failure pattern: latency beat three of four models before reasoning ever became the constraint.