THE SHORT VERSION
● Most bugs per hour → Mo — 250.3 matched in 60 minutes, 17.9%% recall, $2.88 per bug.
● Cheapest per bug → Codex + Playwright MCP — 94.3 matched at $0.56, and the highest precision in the test at 89.0%%.
● Already paying for it → Claude Code — 39.6 matched at $2.79 on Opus 5.5, included in a Pro, Max or Ultra plan you may already hold.
All figures from QA Bench, run by Momentic, whose agent leads it. Exports dated 31 October 2026.
The measured results, side by side
Who ran this matters. QA Bench is Momentic’s own benchmark and Momentic’s own agent tops it. We have not reproduced the runs. The numbers are useful because the suite, the applications and the time budget are published and the competing configurations are named — not because the winner is surprising.
| Configuration | Bugs matched | Recall | Cost per bug | Harness |
|---|
| Mo (Momentic) | 250.3 | 17.9% | $2.88 | Built in |
| Codex, GPT-6.1 Sol | 94.3 | 6.8% | $0.56 | Playwright MCP |
| Codex, GPT-6.1 Terra | 72.7 | 5.4% | $0.52 | Playwright MCP |
| Codex, GPT-6.1 Sol | 69.3 | 5.2% | $1.08 | agent-browser |
| Codex, GPT-6.1 Terra | 53.4 | 4.0% | $0.31 | agent-browser |
| Claude Code, Opus 5.5 | 39.6 | 2.8% | $2.79 | Playwright MCP |
| Claude Code, Opus 5.5 | 33.3 | 2.4% | $2.47 | agent-browser |
| Devin | 27.0 | 1.9% | $3.89 | Built in |
| Claude Code, Sonnet 5.5 | 8.0 | 0.6% | $1.40 | Playwright MCP |
| Claude Code, Sonnet 5.5 | 6.3 | 0.5% | $1.24 | agent-browser |
Read the harness column before the ranking. The same model, Opus 5.5, moved from 39.6 bugs to 33.3 purely by swapping Playwright MCP for agent-browser. Codex on Terra moved from $0.52 to $0.31 per bug the same way, while losing a third of its finds. The browser layer is doing as much work here as the model choice.
Momentic Mo
Finds roughly two and a half times as many bugs per hour as anything else tested, and costs about five times what Codex costs per bug.
Momentic is a Y Combinator W24 company whose agent, Mo, explores a running web application and files what it finds, rather than executing a written test script. Momentic lists Notion, Runway, Webflow, Quora and Retool as customers.
Free tier
2,000 credits a month, no credit card. Runs stop at the cap with no overage billing, which is a real design decision rather than a limitation — you cannot accidentally spend money. Five concurrent browsers, one mobile session, 15-minute session ceilings, 30-day results retention, Google SSO only.
Paid pricing
Pay-as-you-go is $125 a month for 10,000 credits, lifting you to 25 concurrent browsers, five Android and five iOS emulators, five phone numbers, a shared Slack channel and domain claiming. Overage runs at $0.01875 per credit, or you top up another 10,000 credits for $125. Unused credits do not roll over. Enterprise is quoted on test-based pricing and adds SAML SSO, SCIM, audit logs, one-hour mobile sessions and an uptime SLA.
Standout
Recall. 250.3 matched bugs in an hour against a field where second place is 94.3 is not a close contest, and at 30 minutes Mo was already at 153.4 for $2.82 per bug — so the rate holds rather than front-loading.
Honest limitation
Momentic wrote the benchmark. Beyond that, Mo’s own recall swung from 31% on Saleor to 9% on Metabase, so the headline 17.9% is an average across four applications and not a number you should expect on yours.
BEST FOR: pre-release bug bashing where finding more matters more than cost per find
OpenAI Codex
The cheapest bug in the test by a wide margin, and the most accurate report when it does file one.
Codex is not a QA product. It is a coding agent that was pointed at a browser through Playwright MCP, which makes its result here the most interesting one in the table: a general agent with a good browser tool beat two purpose-built QA agents on cost and one of them on volume.
Free tier
Codex rides your ChatGPT subscription rather than carrying its own price. OpenAI’s Codex page states availability in the ChatGPT app but does not publish a Codex-specific rate, so we are not quoting one here — check the plan page for the tier you hold before budgeting a run.
Standout
89.0% precision, the highest of any configuration tested, at $0.56 per matched bug on GPT-6.1 Sol. On the cheaper Terra model with agent-browser it drops to $0.31 per bug. If your bottleneck is triage time rather than discovery, a high-precision agent that files 94 real bugs beats one that files 250 reports you have to sort.
Honest limitation
6.8% recall means it misses more than nine bugs in ten. It is a sampling tool at this setting, not a sweep.
BEST FOR: continuous low-cost checks on every build, where volume is not the goal
Claude Code
Mid-table on bugs found, priced like a seat rather than a run, and probably already in your stack.
Claude Code on Opus 5.5 with Playwright MCP matched 39.6 bugs at $2.79 each. On Sonnet 5.5 it collapsed to 8.0 bugs, which is the clearest signal in the whole benchmark that model choice dominates on this task — same harness, same suite, a fifth of the result.
Pricing
Claude Code is included with Pro at $20 a month, or $200 paid annually, which works out near $17 a month. Max is $100 a month at five times Pro usage and Ultra is $200 a month at ten times Pro weekly usage, with no five-hour limit for verified accounts. Usage on web, desktop, mobile and Claude Code draws from one shared pool, so a long QA run competes with whatever else you were doing that week.
Standout
Marginal cost. If you already hold a Max or Ultra seat for development work, a QA pass is drawn from usage you have bought rather than billed as a separate line.
Honest limitation
That shared pool is the catch. The $2.79 per bug figure is measured spend at API rates; on a subscription the same run costs you headroom you wanted for something else, and the weekly limits are where you feel it.
BEST FOR: teams already on a Max or Ultra seat who want QA without a new vendor
Devin
Last on volume and highest on cost per bug in this test, but it is being asked to do a job it was not built for.
Devin matched 27.0 bugs at $3.89 each. Devin is an autonomous software engineer rather than a QA agent, and QA Bench scores bug discovery, not the fix-and-ship loop Devin is actually sold for. Reading this row as a verdict on Devin would be a mistake.
Pricing
There is a free plan with a light agent quota, unlimited inline edits and unlimited tab completions. Pro is $20 a month with increased quotas and extra usage charged at API pricing. Max is $200 a month. Teams is $80 a month for the team plus $40 per full developer seat. Enterprise is quoted. Devin no longer publishes per-ACU rates on its pricing page — overage is described as API pricing without a per-unit figure.
BEST FOR: fixing the bugs another agent found, not finding them
Which one, for your situation
If you ship weekly and a missed regression is expensive → Mo. You are buying recall, and nothing else in the test comes close. Budget around $125 a month plus overage and expect to tune once you see your own recall rather than the benchmark average.
If you want a check on every pull request → Codex with Playwright MCP. At $0.56 a bug and 89% precision you can afford to run it constantly, and the reports that come back are mostly real.
If you already pay for Max or Ultra → Claude Code on Opus 5.5. Not the best result in the table, but the marginal cost of trying it this week is zero, and Opus 5.5 beats Sonnet 5.5 by five times on this task so do not economise on the model.
If you are a solo founder with no QA budget → Momentic’s free tier. 2,000 credits a month with hard stops and no overage billing is the only option here that cannot surprise you with an invoice.
Running two of them together
The benchmark accidentally describes a stack rather than a winner. Codex is cheap and precise but misses most things. Mo finds most things but costs five times as much per find. Those are complementary failures, not competing ones.
1. Every pull request — Codex with Playwright MCP on the changed surface only. Cheap enough to be automatic, precise enough that the reports do not need triage.
2. Before each release — Mo across the whole application, timed at 60 minutes rather than 30. The gap between 153.4 bugs at 30 minutes and 250.3 at 60 is the half hour that pays for itself.
3. On what comes back — Claude Code or Devin on the fix, not the find. Both are stronger at reading a bug report and producing a patch than at discovering the bug.
4. Quarterly — re-run the comparison on your own application. Mo’s recall varied from 9% to 31% across four codebases in the same benchmark, so the only number that matters is the one you measure on yours.
Frequently asked questions
Which AI agent finds the most bugs?
In QA Bench, Momentic Mo matched 250.3 of 1,396 labelled bugs in a 60-minute budget, a recall of 17.9 percent. Second place was Codex on GPT-6.1 Sol with Playwright MCP at 94.3 bugs, or 6.8 percent. The benchmark was built and run by Momentic.
What does it cost to find one bug with an AI QA agent?
Measured cost per matched bug in QA Bench ranged from $0.31 for Codex on GPT-6.1 Terra with agent-browser to $3.89 for Devin. Momentic Mo came in at $2.88 and Claude Code on Opus 5.5 at $2.79.
Is there a free AI QA tool?
Momentic has a free tier of 2,000 credits a month with no credit card and no overage billing, limited to five concurrent browsers and one mobile session at 15 minutes each. Devin has a free plan with a light agent quota. Claude Code requires a paid Claude plan, starting at $20 a month.
Does Playwright MCP make a difference?
A large one. Claude Code on Opus 5.5 matched 39.6 bugs with Playwright MCP and 33.3 with agent-browser. Codex on GPT-6.1 Sol matched 94.3 with Playwright MCP and 69.3 with agent-browser, at double the cost per bug. The browser layer changed results as much as the model did.
Should I trust a benchmark published by one of the vendors in it?
Treat it as a starting point. QA Bench names its applications, its bug count, its time budget and every competing configuration, which is more disclosure than most vendor benchmarks offer. It is still run by the company whose agent wins, so the honest use is to copy the method onto your own codebase rather than to accept the ranking.
Which model should I pair with a coding agent for QA work?
The stronger one. Claude Code on Opus 5.5 matched 39.6 bugs; the same harness on Sonnet 5.5 matched 8.0. A cheaper model did not save money here, it simply found a fifth as much.