A new test for AI coding agents shows that even the best current model is far from finishing a developer's work on its own. Co-Design Bench, created by researchers from Fudan University, Alibaba Token Hub and other Asian universities, evaluated ten models on 273 tasks taken from real programming sessions — and the best of them, OpenAI's GPT-6 Astra, fully completed only 26.4% of the tasks.
The figure contrasts with another result from the same test: GPT-6 Astra met, on average, 69.9% of each task's acceptance criteria. In practice, the agent usually does most of the work but leaves at least one detail unresolved — and that detail requires human review. It is the difference between “almost done” and “done”.
How the test was built
The tasks were selected from 37,463 deduplicated requests drawn from 26,314 real coding sessions, and each received hidden acceptance criteria written by experts — 2,331 in total. Requests include multimodal inputs such as images, audio, video and documents, and the tasks fall into two tracks: localized multimodal edits, with 100 tasks, and whole-repository edits, with 173 tasks.
Grading uses a “judge-then-refute” protocol: one judge tests the finished application in the browser, and a second verifier tries to overturn every pass, so mock-ups and hard-coded results get no credit. The authors deliberately chose hard cases — requests in which users had to rephrase their ask several times or roll back changes — so 26.4% is a stress-test number, not the success rate on an average task.
The model ranking
Behind GPT-6 Astra came Anthropic's Claude Opus 5, with 18.3% of tasks fully completed, and Alibaba's Qwen3.8-Max, with 16.5%. xAI's Grok 4.6 came last among the ten models, at 6.6%. Whole-repository edits were the hardest: GPT-6 Astra fully solved only 17.9% of them, against 41.0% of localized edits.
Two details stand out. The smaller Qwen3.8-27B climbed to second place on larger repository overhauls, and changing only the agent's execution harness moved its score from 0.560 to 0.749 — which is why the authors recommend always reporting model and harness together. Effort differed too: GLM-5.2 used nearly six times as many tokens as GPT-6 Astra on smaller changes and still scored worse.
Why it matters
The study was published as a preprint on arXiv on 26 October 2026 and has not yet been peer-reviewed. There is also a conflict of interest to note: most of the authors are affiliated with Alibaba — but the top-ranked model is not Alibaba's; it is OpenAI's. The central message is that older benchmarks, on which models score above 99%, no longer distinguish real performance: on tasks taken from everyday work, nearly three in four still need human cleanup.
Read also: GDPx: SemiAnalysis launches a benchmark where the best AI model reaches only 12% on high-value tasks
Sources
- Co-Design Bench: Multimodal Collaborative Coding Evaluation for Agentic Models (arXiv:2610.35138)
- Co-Design Bench official project page
- TechProlonged: Top AI coding agent fully solves just 26.4% of real tasks in new test
Featured image: figure of the Co-Design Bench evaluation pipeline, from the project's official page (codesignbench.github.io).
Transparency: this article was produced with the assistance of artificial intelligence, based on cross-checking public posts and internet sources cited above. We recommend consulting the original sources to verify the data.
By Muse
.svg){kind=logo}
