TRACE: a lower score is an observation, not a diagnosis
An October 8 preprint separates changed behavior from a changed scoring rule. Repeat controls matter as much as the headline.
An October 8 preprint separates changed behavior from a changed scoring rule. Repeat controls matter as much as the headline.
TRACE: a lower score is an observation, not a diagnosis
An October 8 preprint separates changed behavior from a changed scoring rule. Repeat controls matter as much as the headline.
Same work, different verdict
A tool changes its name, but not what it does. If an agent performs the same operations and receives a lower score, the apparent regression may belong to the verifier. TRACE tests that distinction with paired runs, inspection of trajectories and counterfactual rescoring of the unchanged record. These answer different questions: did the whole system’s score move, did behavior change, and did the scoring rule change its judgment? I read the preprint; I did not reproduce its experiments.
Controlled mechanism
In 25 synthetic tasks, renaming tools lowers the careful scripted agent’s mean score by 0.250 even though its operations and simulator quality are unchanged. Mapping aliases back at scoring time removes the entire gap. A second policy has a real behavioral failure under the same mutation. The result is a controlled demonstration of a mechanism, not a general claim that renaming never matters. The deterministic policies repeat identically under seed labels, and each probe family uses one fixed transformation, not a broad paraphrase distribution.
Repeat controls
The public tau²-bench story is less tidy. An initial 30-task study’s one clear effect does not replicate. The main follow-up evaluates four agents and uses 88 previously untested tasks as its main test set, with repeated runs. Renaming or reformatting is equivalent within a prespecified ±0.10 margin for seven of eight agent-change pairs. Identical reruns flip 15-36% of task outcomes. Deliberately misleading tool names lower reward by 0.20-0.44 for every agent, a positive control showing that the pipeline detects consequential changes. Equivalent within a margin does not mean exactly identical or harmless on every task.
My analysis
Two judges disagree on 57% of the same fixed records despite being fairly stable under presentation changes. The paper traces much of that disagreement to grading procedure rather than outcome. My analysis: stability is not validity. A judge can repeat the same answer while measuring a different definition of success. Before a leaderboard comparison, write down whether success means the resulting state, the process, or both, and retain task-level results rather than only a mean.
A workflow to test
My proposed workflow freezes a small synthetic task set, changes one interface feature, repeats the unchanged condition, then rescores the same traces to test the suspected verifier defect. Keep a positive control that should change behavior. The illustrative code below isolates name normalization; it does not implement TRACE or prove benchmark reliability. The paper’s specific tools, tasks, mutations and judges limit generalization. A mean gap alone should trigger investigation, not an automatic model downgrade.
def canonical_calls(calls, aliases):
return [aliases.get(name, name) for name in calls]
assert canonical_calls(["renamed_read"],
{"renamed_read": "read"}) == ["read"]
# Illustrative name normalization only.
# Do not normalize genuinely different operations into equivalence.