Grow the harness, not the prompt: agents that turn failures into code
A September 24 preprint tests whether reusable control code can replace repeated model decisions. The gains are promising, but a benchmark rollback gate is not a safety guarantee.
RESEARCH DIARY · 26 SEP 2026
Picture a browser agent that stops after every click to ask the model the same dull question: search again, filter these results, or back out of a broken page? Asked a few hundred times, that question bloats the prompt and burns tokens on a decision that never changes. A new preprint proposes moving that recurring judgment into code the harness writes for itself. In its tests the trick works, within limits worth reading closely.
Reported facts
- What the paper measured: Two benchmarks and three model sizes, with far fewer model calls than the baseline. These are the authors' own figures.
- The gate tests 50 held-out tasks and cannot certify the open web. Brittle repaired code and overfitting stay open questions.
How the harness grows
The researchers start with a strategy-free scaffold that fixes model and tool interfaces but does not encode a task-solving controller. Execution traces localize failures to functions; an optimizer patches a bounded window of failed tasks together. Every few solved tasks, a held-out success gate checks whether previous capability was damaged. If it was, the repair sequence and task cursor roll back. The deployment model still handles task-specific semantic reasoning.
A concrete example, not a reproduced result
Imagine an agent repeatedly opening web results, discarding pages without evidence and retrying searches. A learned function might standardize result filtering and retry order so the model is called only when a new claim needs interpretation. That is an explanatory example, not a claim that the paper learned this exact function. A page’s embedded instructions remain untrusted; compiling a browser policy must not let page text rewrite authority or user consent.
What the paper measured
On BrowseComp-Plus and WebArena-Verified, the authors used 200 training, 50 held-out gate and 50 final-evaluation tasks per benchmark, with three deployment models from 4B to 120B. They report the top mean success in five of six benchmark-model settings, 76.0–91.8% fewer model calls and 74.4–98.6% lower deployed-agent inference cost than their Tool-Calling baseline. On WebArena-Verified, mean success is 44.7–45.3% across the three models; the 4B Tool-Calling baseline scores 6.7%. These are authors’ benchmark figures, not independent production measurements.
Analysis and limits
The held-out gate blocks regressions on its 50-task set; it cannot certify all future websites, permissions or security cases. Repaired control code may itself be brittle, and an optimizer could overfit a benchmark task family. Cost ratios use the paper’s deployment accounting and chosen baselines, not universal cloud bills. The single-run ablation (36% full method against 18%, 22% and 28% without three components) helps explain mechanism, but is not a replicated safety proof.
What to borrow
Separate recurring control from task-specific judgment. Keep a versioned control program, a bounded edit surface, a held-out regression gate, and a full rollback point. For consequential actions, require independent authority checks outside the learned controller. My read is that this is a useful architecture for repeated task families, not a reason to let an agent rewrite its own permission boundary.
# Illustrative gate pseudocode, not the paper’s production implementation
if gate_success(candidate) < gate_success(checkpoint):
restore(checkpoint)
else:
checkpoint = candidateFAQ
Did the agent learn a new model?
No. The paper edits the executable control harness, not the deployment model weights.
Does the gate make it safe to deploy autonomously?
No. It rejects regressions on its held-out tasks, not every future environment or authorization boundary.
Growing Harness preprint · Full method and results. Version 2 posted 24 Sep 2026; author results, not independently replicated.