Researchers at Shandong University tested three coding-agent harnesses - Claude Code, OpenCode, and mini-SWE-agent, a minimalist open-source harness of roughly 100 lines of Python with a single bash tool and no context compaction - across 492 offline tasks from SWE-bench Verified in isolated containers. Models included Qwen3.6-35B-A3B and Qwen3.8-27B served locally, plus DeepSeek-V4-Flash, GLM-5.3-Flash and HY4-Preview over API, with Claude Opus 5 inside Claude Code as a top-tier reference.
On the 447-task pool, Claude Code and mini-SWE-agent scored within two percentage points of each other, which the researchers judged statistically equivalent inside a five-point margin on both Qwen models. OpenCode trailed by up to 9 points, though about half that gap on Qwen3.8-27B came from runs truncated by a 32,000-token output limit. Cost told a different story: per-task cost differed by up to 3x across harnesses using the same model. On GLM-5.3-Flash a Claude Code trial cost nearly twice a mini-SWE-agent trial and more than three times an OpenCode trial. Initial fixed context was 16,581 tokens for Claude Code, 7,025 for OpenCode and 829 for mini-SWE-agent.
The researchers' own summary is the line to remember: "A vendor's own harness on its own model buys no score advantage." Two caveats sharpen it rather than soften it. On the 45 hardest tasks, a median 13% of tasks flipped between pass and fail purely from switching harness - and rerunning an unchanged setup produced the same median flip rate, so a chunk of harness comparison is noise. And on DeepSeek-V4-Flash the cost ranking reversed entirely. They recommend comparing agents on first-call context size, per-step growth and step count rather than on published scores.
Sources
VentureBeat · The Decoder