The short version
● What MedChemBench tests whether models can reason like a medicinal chemist.
● Size 99 problems, drawn from 117 papers and patents, 761 molecular structures.
● Best score GPT-6.2 Astra, 84.5% of available credit.
● The catch 95.4% at reading structures. About half at choosing what to test next.
OpenAI released MedChemBench on 20 November, a benchmark that asks models to do the thing drug discovery actually consists of: read a molecule's structure, interpret the experimental results around it, and decide which compound is worth testing next.
The result is more interesting than the headline score, because the benchmark separates those steps and the models are wildly uneven across them.
What is in it
| Detail |
|---|
| Problems | 99, drawn from published research |
| Sources | 117 papers and patents |
| Molecules | 761 distinct structures |
| Scoring | Share of available credit, averaged across the 99 problems |
| Models tested | Eleven GPT models |
| Released as | A preprint paper. No dataset download stated |
The scores
| Model | Share of available credit |
|---|
| GPT-6.2 Astra | 84.5% |
| GPT-6 Astra | 83.3% |
| GPT-6.1 Sol | 82.0% |
| GPT-6.1 Terra | 75.1% |
| GPT-5.6 Sol | 58.2% |
Worth flagging: OpenAI's own page is inconsistent on the top figure, giving 84.5% in the introduction and 83.3% at the shared setting in a later section, without reconciling the two. The gap between the newest model and GPT-5.6 Sol is the durable finding either way — 26 points in roughly a generation.
Where it falls apart, and why that is the story
Astra earned 95.4% of the available credit for reading molecular structures. On selection and numerical prediction — choosing which compound to make next, and predicting how it will behave — it earned about half.
That is the gap between a very good reader and a working chemist. Structure interpretation is pattern recognition over a well-documented notation. Deciding what to synthesise next is a judgement under uncertainty about an experiment nobody has run. The benchmark shows the first is close to solved and the second is not.
OpenAI's own limitations, which are unusually direct
The post states that the results "do not establish whether models can discover new drugs or improve laboratory decisions". It also says prior exposure to the published sources cannot be ruled out — these problems come from papers the models may have trained on — and that each response was a single attempt with no tools, no feedback and no new experiments.
Taken together that is a benchmark measuring recall-plus-reasoning on solved problems, not discovery. OpenAI says so itself, which is more than most benchmark releases manage.
FAQ
What is MedChemBench?
An OpenAI benchmark of 99 medicinal chemistry problems, drawn from 117 papers and patents and covering 761 molecular structures, testing whether models can read structures, interpret results and choose what to test next.
Which model scored highest?
GPT-6.2 Astra, with 84.5% of the available credit. OpenAI's page also cites 83.3% for the same model at the shared setting and does not reconcile the two.
Can AI discover drugs now?
No, and OpenAI says so directly: the results do not establish whether models can discover new drugs or improve laboratory decisions.
What are models bad at here?
Choosing which compound to test next, and predicting numerical properties. The top model earned about half the credit on both, against 95.4% for reading structures.
Can I download MedChemBench?
No download location is stated. OpenAI published a preprint paper describing it.