Researchers at ByteDance's Seed team published HarnessDev, a benchmark that evaluates whether large language models (LLMs) can design and improve their own agent harness — the execution loop, tools, context management and verification logic that surrounds a model during agentic tasks — rather than judging only the answers a model produces. The paper appeared on arXiv on September 1, 2026, with co-authors from the Singapore University of Technology and Design and the Georgia Institute of Technology, and was covered by MarkTechPost on September 11.
The benchmark runs in two stages. In a creation phase, six models — Opus 4.8, GPT-5.5, Gemini 3.1 Pro, DeepSeek V4 Pro, Qwen 3.7 Max and Seed 2.0 Pro — build a harness from minimal starting components. In an evolution phase, each model then revises its own harness using feedback from execution results. The researchers tested the resulting harnesses across 2,207 task instances drawn from five existing benchmarks: BrowseComp (1,266 web-research tasks), SWE-bench Pro (731 software-engineering instances), Terminal-Bench 2.1 (89 command-line tasks), MLE-bench (75 machine-learning instances) and EQ-Bench3 (46 writing instances).
Strong on writing, weak on search
Self-built harnesses matched or exceeded human-engineered reference harnesses on writing tasks, where Opus 4.8 scored 84.6 on EQ-Bench3 against a reference of 83.7, and on machine-learning experimentation, where it scored 32.9 against 24.0. They lagged well behind the references elsewhere: 52.6 against 92.2 on the BrowseComp search benchmark, and 68.8 against 88.8 on Terminal-Bench 2.1. Averaged over the whole evaluation, the best self-built harness reached 67.8 against 86.2 for the human-engineered reference.
The most notable finding concerned the evolution phase. Of 64 harness modifications the models attempted, only 34 — 53.1 percent — produced feedback scores and held-out scores that moved in the same direction, meaning roughly half the self-directed changes did not generalise to tasks the models had not seen during development. Harnesses were also closely tied to the model executing them: one Opus 4.8 harness scored 69.3 on SWE-bench Pro but fell to 33.0 once a different model ran it, and a duplicate-query rate of 10.1 percent rose to 88.2 percent after the executor was switched. Self-reported success proved unreliable, with only 48 of 99 claimed successes passing verification.
Why it matters
The results bear on a live debate about AI systems that improve their own tooling: the paper suggests such self-improvement is currently narrow, inconsistent across domains and tied to the specific model running the harness rather than transferring cleanly between models. Cost varied widely too — GPT-5.5 reached a 19.1 medal rate on MLE-bench using 29.3 million tokens, while DeepSeek V4 reached 19.6 using 208.4 million. For developers building autonomous coding or research agents, that implies a harness tuned for one underlying model may not carry over if the model is swapped.