Coding agent harnesses: pass rates flat, costs can triple

What happened

A controlled study has found that switching between coding-agent harnesses, the frameworks that wrap a language model and direct it through software tasks, barely changes how often agents pass their tests. The tools examined include Claude Code, OpenCode and mini-SWE-agent. The differences between harnesses were no larger than the variation seen by simply rerunning the same setup.

The cost picture is less even. According to The Decoder, which reported the findings, cost per task can roughly triple between setups. The main driver is the system prompt, which is resent with every step an agent takes, so its overhead compounds over longer tasks.

Why it matters

For organisations deploying AI agents, this shifts the question from which harness to buy towards what each task actually costs. Feature-rich tooling may add little to success rates while raising running costs, which matters most where agents handle high volumes of work, such as internal service desks, code review or customer-facing automation. Pass rate alone is an incomplete measure of value.

The finding also points to a design issue that extends beyond coding. Any agent that carries a large, repeated instruction set pays for that context at every step. Teams building AI-enabled services will need to treat prompts and context as operating costs, and evaluate them with the same rigour as they would any other part of the delivery model.

The Renascence take

Most buyers will be drawn to the visible layer, the named tool and its feature list, because it is easy to compare and easy to explain to leadership. The study suggests that layer is largely cosmetic. The cost sits in the invisible, recurring part of the system, which is far less likely to be scrutinised.

Teams are paying for the harness they can see and ignoring the prompt they resend thousands of times a day. That is the same salience bias behavioural economists observe in subscriptions, where the headline price gets the attention and the compounding fees do not. A customer-obsessed operator should measure cost per successful outcome rather than pass rate, audit the system prompt as if it were a recurring fee, and treat context length as a budget to be managed. Choose the tool last, not first.