A successful agent run is not a reliable configuration

Reported facts: what was varied

The October 1 preprint "Agents are systems, not models: Rethinking agentic evaluation" studies four scientific tasks in astrophysics and genomics. The coding agent must find and operate a published specialist model. The authors vary task information, reasoning, verification instructions, runtime budget and backbone model. The main grid has 432 configurations per task, each repeated five times: 8,640 runs. The released collection contains 18,240 trajectories including additional experiments.

Information has four levels: no extra help, the specialist's identity, its loading and calling interface, and a full task protocol. This is more concrete than "write a better prompt." It tests whether the agent knows which tool exists and how to use it. Among genuine attempts, roughly 54% of score variance comes from repeating the same configuration rather than differences between configurations. That is not a 54% failure rate.

The headline has a model-specific exception

In the main experiments, information has the largest effect on the score, ahead of model and budget. More information also lowers cost and runtime and improves calibration. Extra time does not help every configuration: it can raise completion without raising quality when the agent lacks enough information to use the time productively.

The closed-model follow-up changes the ordering. Budget ranks first for Claude Sonnet 5, while information ranks first for the open-weight families tested. The paper does not establish that documentation always beats a bigger model. It establishes that the useful configuration depends on the model and task. A setting tuned for one model should be tested again when the model changes.

Verification: a request is different from an instrument

Prompting the agent to verify its answer has the smallest score effect among the five main axes. The authors then add an oracle tool that can score a current submission against the task reference. It improves scores and calibration for two tested models across all four tasks, but increases runtime and cost. Reference-based verification rises from about 19% to 22% across prompt-strength levels without the oracle; with it, that behavior roughly triples.

An oracle is a stronger instrument than most production checks. The authors inspect oracle-driven answer search, report it in 4.4% of runs, and say removing those runs barely changes the result. These are the authors' findings, not measurements I reproduced. A validation set in real work can be incomplete, stale or available to the agent in a way that encourages overfitting.

My analysis: make the comparison repeatable

I would start with one narrowly defined task and two conditions: an ordinary brief, and the same brief plus the correct tool identity, interface and task protocol. Keep model version, allowed actions and time cap fixed. Repeat each condition at least five times as a small diagnostic, not a statistically sufficient proof. Record completion, reference score, wall time, billed cost and the agent's own predicted score separately.

Use a separate checker whose reference answers are not written by the same agent. Then compare prompted checking with access to that checker. Preserve failures and timeouts in the ledger; filtering them away can make a slow configuration look excellent. This is a proposed workflow, not executed code or a new benchmark. The practical question is whether the change makes the distribution better, rather than whether it produces one attractive demonstration.

What I would not conclude

Four tasks across two domains cannot settle all agent work. Memory and multi-agent coordination were not varied. The reported gains and rankings are bounded by these tasks, models and scoring rules. The dedicated oracle is not a promise that any added checker will help. I read the paper and repository today; I did not run the authors' code. My takeaway is to test the whole configuration and report its repeated outcomes, not to crown a universal prompt or model.

Illustrative Python ledger, not an executed evaluation

from dataclasses import dataclass

@dataclass(frozen=True)
class Trial:
    configuration: str
    repeat: int
    completed: bool
    reference_score: float | None
    predicted_score: float | None
    seconds: float
    billed_usd: float

# Preserve timeouts as completed=False; do not delete them.
# Populate reference_score only from the independent checker.