VERSE: an optimizer that tests its own edits before it submits them

What the paper claims

The setup has two roles. The executor is a frozen model that attempts coding tasks inside a harness. The optimizer reads failed trajectories and edits the executor’s harness over several rounds. Earlier systems keep the optimizer fixed. VERSE lets it revise its own prompts, skills, tools, hooks and notes, while the weights of both models stay frozen.

The first study is the interesting one. With a Qwen3.8-Flash-Next optimizer and a Qwen3.8-27B executor, adding self-evolution to Meta-Harness made things worse: no round beat the starting harness on validation. The authors’ reading is that the optimizer wrote untested lessons into itself and could not tell which were wrong. Give it a tool to run a draft on one training task first, and the same setup reached 41.05% test accuracy, the highest in that table.

How VERSE works

Before an edit is submitted, the optimizer can replay a failure, cut a failed trajectory down to the few steps that matter, and perturb a suspected step to see whether the outcome changes. Between rounds it keeps a ledger of which tasks were fixed, which regressed and which fixes were later undone. In a second study across five executors, the self-evolving optimizer built 51 artifacts, and they fell into four kinds: attribution of failure causes, verification, a training audit, and workflow rules such as budget reminders.

A sketch of the loop

# Illustration of verify-before-submit. Not the authors' code and not run.
def improve_harness(harness, failures, budget):
    for round_no in range(budget):
        cause = rank_causes(failures)            # attribution
        draft = edit(harness, cause)
        passed_before = [t for t in tasks if run(harness, t)]
        ok_target = run(draft, cause.task)       # replay the failure
        ok_old = all(run(draft, t) for t in passed_before[:5])  # regressions
        if ok_target and ok_old:
            return draft
    return harness

The numbers

The paper’s summary holds: VERSE with self-evolution beats its own unequipped baseline in all 16 comparisons of the table, by 0.6 to 10.3 points. Reading the table closely adds a qualification. Among the four hosts, the best in-distribution harness comes from VERSE without self-evolution on three of them, and the best out-of-distribution harness comes from the self-evolving version on two. So the verification tools do most of the work in distribution, and the case for self-evolution rests mainly on the newer, multi-language tasks.

Where the evidence is thin

Each row is one evolution run. The error bars are the spread over three test evaluations of that run, not over independent runs. The in-distribution test set has 108 tasks, so one task is about 0.9 points, and several gaps in the table are two or three tasks. All training and validation tasks are Python, and the 107 out-of-distribution tasks are newer ones with 87 in Go, Java, Rust or TypeScript. The authors also say the optimizer’s verification tools are fixed, every action is logged, and the executor runs without network access. That bounds the scope of self-modification in their setup, and it is the part a practitioner would have to rebuild.

Who should care

Teams that tune an agent harness by hand or with an optimizer. The practical lesson survives the caveats: give an automatic editor a way to run its change and check that already-passing tasks still pass, before you let it keep that change. The paper notes that the fixed optimizer often tested only tasks the current harness fails, which cannot show a regression.