THE RESULT
● GPT-6 Astra: 49.2 percent on full-application reconstruction.
● Claude Opus 5: 28.8 percent on the same task.
● The benchmark: Microsoft Research, 17 September 2026. Nine frontier coding agents, 26 applications, 4,063 tasks built from 1,975 replay-verified behaviours.
● The finding nobody is quoting: both fall apart with depth. One agent drops from 100 percent to 64 percent, another from 96 percent to 32 percent, as restoration depth goes from one layer to eight.
What ProgramDistill actually asks a model to do
Almost every coding benchmark hands the model an instruction. ProgramDistill does not. It gives the agent a working reference application and an incomplete copy, and asks it to infer the missing behaviour by interacting with the working version, then implement it.
That is much closer to the job. Most real engineering work is not "write a function that does X" — it is "make this behave like that thing over there," where nobody has written down what "that" does.
| Model |
Full-application reconstruction |
Gap |
| GPT-6 Astra | 49.2% | — |
| Claude Opus 5 | 28.8% | 20.4 points behind |
| Seven other frontier agents | Tested, scores not broken out in the summary | — |
The depth number is the real story
A 20-POINT HEADLINE GAP, AND BOTH MODELS FAIL THE SAME WAY
On partial reconstruction the paper reports one agent falling from 100 percent to 64.0 percent, and another from 96 percent to 32 percent, as restoration depth increases from one layer to eight.
Shallow work is close to solved. Layered work is not. A two-thirds collapse over eight layers is the number to carry into any decision about handing an agent a multi-step task unattended.
This is the pattern that keeps showing up and keeps getting ignored in launch coverage. Single-step accuracy is high and rising. Accuracy compounded over a chain of dependent steps is not.
What this does and does not tell you
- It is one benchmark, from one lab, on web applications. Twenty-six apps is a reasonable spread and not a universe. It says little about systems programming, data pipelines, or anything without an observable UI.
- Microsoft published it. Worth stating plainly. The result favours the model Microsoft is closest to commercially. That is not evidence of anything improper, and it is a fact a reader should have.
- Reconstruction is not authoring. Inferring behaviour from a working reference is a specific skill. A model that is better at it is not automatically better at greenfield work.
- Nobody scored above 50 percent. The leader fails more than half of these tasks. That is the honest headline.
How to use this when picking a coding agent
If you are cloning or porting behaviour from an existing system — a rewrite, a migration, matching a legacy service — this is the closest published proxy for that work, and GPT-6 Astra leads it clearly.
If you are chaining more than a few dependent steps, neither model is reliable unattended. Both degrade sharply with depth. Break the work into shallow pieces and check between them.
If you are choosing on price rather than capability, this benchmark is not your input. See our Codex pricing breakdown and Cursor pricing breakdown for what these agents actually cost to run.
Sources
FAQ
What is ProgramDistill?
A coding agent benchmark from Microsoft Research, posted 17 September 2026. It gives an agent a working reference application and an incomplete copy, and scores whether the agent can infer the missing behaviour and implement it. The dataset is 1,975 replay-verified behaviours across 26 applications, producing 4,063 tasks.
Which model won?
GPT-6 Astra, at 49.2 percent on full-application reconstruction. Claude Opus 5 scored 28.8 percent on the same task. Nine frontier coding agents were tested in total.
Is a 49.2 percent score good?
It is the best of nine frontier agents and it still fails more than half the tasks. Treat it as a ranking, not a readiness signal.
What happens as tasks get deeper?
Performance drops sharply. The paper reports one agent going from 100 percent to 64.0 percent and another from 96 percent to 32 percent as restoration depth increases from one layer to eight. Shallow tasks are close to solved; layered ones are not.
Does Microsoft publishing it affect the result?
The benchmark is public and the paper is a preprint anyone can check, so the method is auditable. It is still worth knowing that the result favours the model Microsoft is closest to commercially.
Should this change which coding agent I use?
Only if your work resembles the task. For cloning or porting behaviour from an existing system, GPT-6 Astra leads clearly. For greenfield authoring, or for anything priced-constrained, this benchmark is not the right input.