Which failure to fix next: reading ActiveSaddler
A harness repairs a shell-quoting bug, then spends its next ten rollouts on tasks it already passes.
A harness repairs a shell-quoting bug, then spends its next ten rollouts on tasks it already passes. Nothing is broken in any single step, yet the budget is wasted. That scheduling problem is the center of the ActiveSaddler preprint posted to arXiv on October 1.
The short version
Reported research: ActiveSaddler chooses which training scenarios drive automated harness updates. It treats failure patterns as arms of a non-stationary bandit, estimates learning progress for each, and balances revisiting known weaknesses with exploring new scenarios. On GAIA2 it reports 59.8% test Pass@1 against 55.4% for the same optimizer with a fixed shuffled order. On Terminal-Bench 2.0 it reports 80.0% against 72.5%. ↗
The numbers, and what they cover
The GAIA2 test split has 300 tasks and the Terminal-Bench 2.0 split has 40. Scores are means over three test-time executions: 59.8 ± 1.0 on GAIA2 (default agent 53.6, GEPA 54.2, Meta-Harness 54.2) and 80.0 ± 2.5 on Terminal-Bench 2.0 (manual Terminus-KIRA 69.2). The headline gains of 4.4 and 7.5 points are against the same optimizer with a scenario order fixed before optimization. The authors also report an independent GAIA2 run that again beats the fixed orders, and a +3.0 point lift when the method is applied to GEPA (54.2% to 57.2%). ↗
Forty test tasks
This is my analysis, not the paper's claim. On 40 tasks one task is 2.5 points, so 7.5 points is the size of three tasks averaged over runs. That is a useful signal and a thin one. The 300-task GAIA2 result is a steadier basis, and its 4.4 point gain is the number I would plan around. Neither tells us how the method behaves on a private codebase, a different base model or a harness that already works well. ↗
How failures become arms
Each arm is a failure pattern abstracted from failed trajectories, for example a repeated command-construction error, not a task category or a single task. The ablations make the choice matter: swapping in category arms lowers Terminal-Bench 2.0 by 10.8 points and scenario arms by 6.7, and removing the arm prioritizer or the exploration controller each falls back toward the fixed-order result. My reading: the gain comes from remembering that a patch failed and returning to the pattern, while still spending some budget on scenarios that might reveal new patterns.
A toy test I wrote
I wrote an invented bookkeeping example and executed it locally: seven fixable failures across four patterns, six rollouts. A preset order that starts with an already-solved pattern repairs 3; always targeting the largest unresolved pattern repairs 6. It also asserts that one task on a 40-task split moves a score by 2.5 points. These inputs are made up. They illustrate the allocation idea and are not evidence about ActiveSaddler.
Limits and security
The authors state the work is a research exploration, run on public and synthetic benchmark data, and not evaluated as a production system or for security. They note that a generated harness update can change prompts, tool interfaces, command construction or control flow without touching model weights, and recommend least-privilege tools, sandboxing, adversarial testing and explicit review for changes that touch permissions. The paper says project code and a website will be available; I did not verify a release, download anything or run the method. This is an arXiv v1 preprint.
The toy test, in code
# Invented toy: two ways to spend 6 rollouts on failure patterns.
remaining = {"quoting": 4, "paths": 2, "timeouts": 1, "stale-docs": 0}
def fixed(order, budget):
left = dict(remaining); fixed_n = 0
for name in order[:budget]: # one pass in a preset order
if left[name]:
left[name] -= 1; fixed_n += 1
return fixed_n
def adaptive(budget):
left = dict(remaining); fixed_n = 0
for _ in range(budget): # always target the largest unresolved pattern
name = max(left, key=left.get)
if left[name]:
left[name] -= 1; fixed_n += 1
return fixed_n
order = ["stale-docs", "stale-docs", "timeouts", "paths", "stale-docs", "quoting"]
a, b = fixed(order, 6), adaptive(6)
assert (a, b) == (3, 6)
print("preset order repaired", a, "of 6; adaptive repaired", b, "of 6")
# 1 test task on a 40-task split moves the score by 2.5 points.
assert 100 / 40 == 2.5