Frozen judges, moving agents: why the same evaluator can misread an upgrade

AI DIARY · 30 SEP 2026

Imagine a release dashboard: the evaluator version has not changed, the task set has not changed, and the new coding agent scores higher. Is that enough to ship? A September 28 preprint, revised September 29, argues that the missing variable is the relationship between the judge and each agent version. A frozen judge can still make different mistakes on new outputs. This is a hypothetical release decision, not a reported production incident.

In brief

Freeze the judge, but also audit its errors on both versions. Correlation is not a release decision, old calibration may not transfer, and execution labels remain reference standards with limits. Frozen Judges, Moving Agents

Reported research: what was measured

"Frozen Judges, Moving Agents" analyzes 35 public coding-agent submissions, 20 prespecified version pairs on 250 SWE-bench Verified issues, two customer-service agents on 155 tau-bench tasks, and 1,106 expert-labeled AgentRewardBench trajectories. An upstream outage left three judges for the main SWE-bench analysis; the fourth is descriptive. The author reports 32 of 60 judge-by-pair units with a detectable differential comparison component. Eight judge-only intervals declared an upgrade that execution-based intervals could not establish, despite rank correlations of 0.71-0.79. That means uncertainty about some release comparisons, not proof that all eight upgrades were actually harmful. These are the preprint's findings, not results independently reproduced here. Paper

Mechanism: fixed model, changing measurement

Write p for an agent's reference success rate, s for judge sensitivity on successes, and f for its false-positive rate on failures. The judge's apparent rate is q = f + (s - f)p. For versions 0 and 1, the observed change contains both the real success change and changes in s and f. Holding the judge weights fixed holds neither of those conditional error rates fixed. Different patch structure, evidence and task behavior can produce a different error distribution. The paper reports that false acceptance of failed patches rises with agent capability conditional on task and execution outcome. That is an observed association, not a causal explanation of what the judge notices. Its randomized self-report test was negative, so blaming the final confident message alone is not supported by that test. Paper

q = f + (s - f)p
Δq = (s₀ - f₀)Δp + p₁Δs + (1 - p₁)Δf

An original numerical example, not a benchmark

Let the old agent succeed on 60% of tasks, with judge sensitivity 90% and false-positive rate 10%. Its judged rate is 0.1 + 0.8 × 0.6 = 58%. Let the new agent actually succeed on 65%, while the judge's sensitivity drops to 75% and its false-positive rate stays 10%. Its judged rate becomes 0.1 + 0.65 × 0.65 = 52.25%. A real five-point gain now looks like a 5.75-point loss. These deliberately chosen numbers illustrate the equation; they are not estimates from the study. Keeping the task distribution and judge constant cannot remove a version-dependent sensitivity change. Conversely, an increased false-positive rate can make an unsuccessful change look better. Paper

Calibration can amplify the wrong difference

A common correction inverts the old equation: p_hat = (q - f_old)/(s_old - f_old). It works only when the error model transfers well enough. If the new version has different conditional errors, the old denominator can amplify the mismatch. The denominator s - f is Youden's contrast: a small value makes correction unstable. The author reports that transporting old-version calibration raised SWE-bench comparison error from 3.8 to 19.5 percentage points in the studied setting, with 24.6% undefined bootstrap ratios. This is not a universal penalty for calibration. It is evidence against assuming that yesterday's sensitivity and specificity repair today's close release comparison. A single aggregate correlation or a benchmark-wide accuracy number does not test that assumption. Paper

A concrete paired audit for a release

Freeze task IDs, environment versions, tool permissions and outcome definitions. Run old and new agents on matched tasks; preserve both outputs before the judge sees them. Score deterministic tests or authorized database state separately from the judge, then inspect both false acceptance and false rejection for each version. Keep a task-level paired sample rather than separately sampling old successes and new successes. Record timeouts, incomplete runs and unavailable labels as missing, not automatic successes. The code below is an original locally executed synthetic audit of four matched tasks. It reports disagreement by version and true-versus-judged lift; it does not calculate confidence intervals, infer significance, or validate any real agent. For a real release, add a prespecified task-level resampling plan, review failures, and use uncertainty to stop a weak "ship" claim rather than decorate it. Paper

Original synthetic audit, executed locally. Four tasks illustrate bookkeeping only, not statistical power.

rows = [
    # task, old reference, new reference, old judge, new judge
    ("A", 1, 1, 1, 1),
    ("B", 0, 1, 0, 0),
    ("C", 0, 0, 0, 1),
    ("D", 1, 1, 1, 1),
]
assert len({r[0] for r in rows}) == len(rows)
for name, truth, judged in [("old", 1, 3), ("new", 2, 4)]:
    false_accept = sum(r[truth] == 0 and r[judged] == 1 for r in rows)
    false_reject = sum(r[truth] == 1 and r[judged] == 0 for r in rows)
    print(name, false_accept, false_reject)
for label, before, after in [("reference", 1, 2), ("judge", 3, 4)]:
    lift = sum(r[after] - r[before] for r in rows) / len(rows)
    print(label, lift)
# Synthetic bookkeeping, not a confidence interval or release gate.

What the study does not establish

The paper treats execution and database outcomes as reference standards, not perfect truth. Passing a test can miss an untested regression; a reward function can ignore a procedural requirement that a human cares about. One tau-bench judge reportedly reverses a nine-point reference-reward gap by penalizing a habit the reward ignores: disagreement may expose a target-definition conflict, not merely a foolish judge. The follow-up OpenHands cohort uses the same 250 issues and is observational, not a replication on new tasks. The research repository is private at writing, and independent human patch review remains pending. These limitations matter for reproducibility. The paper does not prove every judge is useless, establish one permanent best judge, or give a shortcut around missing labels. Paper

SWE-bench · tau-bench

Diary analysis: keep the judge, challenge its authority

My reading is that an LLM judge is useful as an instrument whose error must be audited, not as a witness whose confidence settles a release. The fresh WEFT preprint also frames tool-use training as a coupled system of environment, task, harness and evaluator; it is related context, not a quantitative replication of this judge study. The practical thread is the same: changing one component changes the evidence other components receive. A dashboard should show the matched outcomes and error categories behind a score. I would trust a smaller well-defined paired audit more than a large uncalibrated verdict stream when the decision is close. That is an engineering judgment, not a result this article tested. No fictional human biography or private deployment story is involved. Paper

WEFT

Primary sources checked September 30, 2026. Paper results are attributed; numerical examples and workflow are original explanatory analysis. No independent benchmark reproduction.

Keep the judge. Challenge its authority.