Skip to news

Co-Design Bench Finds Top AI Coding Model Meets 69.9% of Criteria but Fully Completes Only 26.4% of 273 Real-World Tasks

A 16-author arXiv preprint built from real user coding sessions measures the gap between partial success and shippable work, with the best of ten models needing human repair on nearly three tasks in four.

By THE COLDAI TIMES desk (AI-assisted)Editor: Shayan SalehiPublished 5 min read1,013 words
Contextual file photo: Alibaba Group's headquarters campus in Hangzhou; the benchmark's authors are linked to Alibaba's Qwen team by some reports
Contextual file photo: Alibaba Group's headquarters campus in Hangzhou; the benchmark's authors are linked to Alibaba's Qwen team by some reports — Thomas LOMBARD, designed by HASSELL (architects)[1] · Alibaba group Headquarters (cropped).jpg · Wikimedia Commons · CC BY-SA 3.0

Key facts

  • In the Co-Design Bench paper posted to arXiv on 26 October 2026, the best of ten model endpoints satisfied an average of 69.9% of criteria per task but fully completed only 26.4% of the 273 tasks.[1][3]
  • The 273 tasks were curated from 37,463 deduplicated queries across 26,314 real user coding sessions, and expert authors wrote 2,331 acceptance criteria to grade them.[3][1]
  • The Decoder reported on 8 November 2026 that GPT-6 Astra was the top model, fully solving 41% of multimodal tasks and just under 18% of repository tasks.[2]
  • All ten tested model endpoints scored lower on repository edits combining three or four kinds of change than on those combining one or two.[1][4]
  • Grading uses no gold implementation: a judge model scores each criterion and a separate verifier challenges every passing verdict using source code and browser evidence.[1][4][2]

The best of ten AI coding models tested on Co-Design Bench met an average of 69.9% of acceptance criteria per task but fully completed only 26.4% of 273 tasks, according to a paper posted to arXiv on 26 October 2026 and reported by The Decoder on 8 November. The benchmark, built from real user coding sessions by a 16-author team, is designed to show the gap between "mostly done" and "ready to ship" that older coding tests, where models score above 99% on some, tend to hide.

What is Co-Design Bench?

The arXiv paper, titled "Co-Design Bench: Multimodal Collaborative Coding Evaluation for Agentic Models," lists 16 authors led by Xilin Wei and Shengyuan Ding. The arXiv page does not list affiliations. Sources describe the origin differently: The Decoder says the work comes from researchers at Alibaba and several Asian universities, while ExplainX attributes it to the Qwen team and collaborators. Alibaba develops the Qwen models, which matters for how the results are read, as discussed below.

The test asks an agent to turn a real user's change request into a working revision of an existing application. According to the AI Weekly summary, the 273 tasks were curated from 37,463 deduplicated queries across 26,314 real user coding sessions, and expert authors wrote 2,331 acceptance criteria to judge the results. The tasks split into two tracks. Multimodal Editing covers localised edits that may include images, audio, video or documents as input. Repository Editing covers coordinated revisions across a codebase.

What did the results show?

The headline result is a gap. In the paper's abstract, the strongest model satisfied an average of 69.9% of criteria per task, yet fully completed only 26.4% of tasks, a difference of 43.5 percentage points, per AI Weekly. ExplainX puts the practical meaning plainly: most of a request usually gets done, but one remaining defect can block shipping.

The Decoder reports that the top model was GPT-6 Astra, which fully solved 41% of multimodal tasks and just under 18% of repository tasks. It names Claude Opus 5 and Qwen3.8-Max as the next two, and Grok 4.6 as last. The Decoder also reports that GLM-5.2 used nearly six times as many tokens as GPT-6 Astra on smaller changes while scoring significantly worse. AI Weekly's summary does not name the models, and the arXiv abstract page reports only the top score, so the model rankings and the per-track figures rest on The Decoder's reading of the paper.

Complexity hurt every model. The paper says all ten endpoints scored lower on repository edits that combined three or four kinds of change than on those that combined one or two. The Decoder lists the change types as layout, design, interaction and content. It also reports that the harness, the environment in which an agent reads, writes and runs code, affected results as much as the model did.

How are tasks graded without a reference answer?

Co-Design Bench uses no gold implementation, unit tests or target screenshots. According to the arXiv abstract, a judge model scores each task on three dimensions: grounded understanding, UI transformation and functional implementation. A separate verifier then challenges positive verdicts using source code and browser evidence, a "judge-then-refute" process meant to catch stubs and hardcoded results. The Decoder describes the second judge as re-checking every passing verdict for mock-ups.

That design addresses a known weakness of coding benchmarks. The Decoder notes related findings: a METR study found that project leads accepted AI-generated code about 24 percentage points less often than SWE-bench Verified scores would suggest, and flaws in test environments can let agents pass tasks without solving them. A separate study it cites found little measurable benefit from elaborate harnesses despite up to three times the token use. The Times has not reviewed those studies and reports them as The Decoder characterises them.

Why does it matter for software teams?

Partial credit and full success lead to different decisions. A tool that gets 70% of a change right still hands a developer a diff to review, repair and re-test, which is the cleanup burden The Decoder's headline refers to. For organisations budgeting around agentic coding, the benchmark suggests measuring how often a task is finished without human repair, not how many requirements it touches. ExplainX offers similar practical takeaways: report both the criterion pass rate and the fully-solved rate, split large requests into smaller checkable steps, and spot-check judge verdicts against saved screenshots and logs.

The results also bear on competition among model makers. OpenAI's GPT-6 Astra leads on the paper's strict measure, but ExplainX notes that the authors build Qwen models and that its article did not rerun the benchmark. The Decoder reports that Qwen3.8-27B ranks second on larger overhauls, though the secondary summaries give differing scores for it, so the Times does not repeat them. The authors themselves say, per ExplainX, that "Rankings vary with the editing setting."

What we don't know yet

Several caveats limit how far the numbers travel. The Decoder says each task uses one fixed request, so follow-up questions and mid-task corrections, which real developers use constantly, are not tested. LLM judges can still make errors, according to both The Decoder and ExplainX. ExplainX adds that the tasks focus on web-style applications, so results may not transfer to backend or embedded work, and that the paper is recent and may be revised. The Times has read the arXiv abstract page and three secondary accounts, not the full paper, and has not rerun the benchmark. Affiliations are unconfirmed on the arXiv page, and the model names behind the ten endpoints appear only in The Decoder's account.

What happens next?

The paper is a preprint, so it has not been peer reviewed. Independent reruns, particularly by labs whose models are ranked, would show whether the 26.4% figure holds under different harnesses and judges. Developers choosing between coding agents now have a benchmark that rewards finished work rather than near misses; whether vendors begin reporting a fully-solved rate beside their usual scores will indicate whether it changes how the market measures progress.

Add The ColdAI Times as a preferred source on Google

Related stories