THE VERDICT
● The test: could this commitment produce a finding the company did not want?
● If yes, it costs something and it means something.
● If no, it is a description of work already done, graded by the people who did it.
Four kinds, ranked by what they cost
| Kind | Example | Can it produce unwanted findings? |
|---|
| External audit with access | Anthropic giving METR transcripts | Yes, if publication is independent |
| Disclosure against interest | Anthropic publishing four breaches | Already did |
| Capability gating | OpenAI routing Astra cyber via Daybreak | Costs revenue. Enforcement unpublished |
| Framework document | Most published safety policies | Rarely |
THE DETAIL THAT DECIDES THE TOP ROW
External access only counts if the auditor publishes independently. Access with a review veto is consultation wearing an audit costume.
That detail has not been published for the METR arrangement, and it is the first thing to ask about any similar commitment.
Questions to ask any vendor
- Has an external body had access? To what, and could they publish without approval?
- Have you disclosed anything against your own interest? If not, the safety documentation is marketing.
- What have you declined to ship? A lab that has never withheld anything has never had to choose.
- How is any gating enforced technically? Policy and control are different, and only one survives an incident.
- What happens to my data in an audit? Production transcripts contain what your users typed.
FAQ
Is a published safety framework worthless?
No, but it is the weakest of the four. It describes intent rather than demonstrating a cost, and every lab has one.
What makes external audit different?
It can produce findings the company did not want, provided the auditor publishes independently. Without that, it is consultation.
Which labs have done the strongest version?
Anthropic has both disclosed breaches against its own interest and granted external transcript access. OpenAI has gated a capability at revenue cost. Both are more than a document, and neither has published full enforcement detail.