The explanation is not the repair: reading the Agent Error Dataset
A failed agent run can leave an excellent explanation and the same broken task.
A failed agent run can leave an excellent explanation and the same broken task. That gap is the interesting part of the Agent Error Dataset preprint. Today I read it as a study of evidence: which decision could change, what the agent could see at that decision, and whether changing it actually helps. A convincing diagnosis is a proposal, not a recovered outcome. This opening describes a general engineering problem, not a private deployment or a reported incident.
The short version
Reported research: AED contains 50,228 error-diagnosis pairs from 9,961 source tasks across 33 environments, 19 harness families and 23 policy models. Those are coverage counts, not 50,228 independent successful repairs. The authors preserve traces and execution metadata. Their five-stage Agentic Error-to-Training pipeline collects natural failures, creates diagnoses and proposed corrections, checks trace support, replays where possible, and builds separate training views. A diagnosis view can exist without a passing replay; a recovery target requires an executed passing continuation. ↗
Matched replay: what the improvement means
The strongest correction result uses 3,062 matched replay pairs. A first proposed correction passes the verifier in 51.1% of cases, versus 18.4% for a fresh retry of the original action at the same checkpoint. The reported gain is 32.7 percentage points, with a task-clustered 95% interval of 28.4-37.0. Matching the checkpoint, policy, harness, budget and verifier makes the comparison more useful than comparing a repaired run with yesterday's final failure. It does not show that the chosen step is the unique root cause, or that an agent can find it autonomously. ↗
Teacher agreement is a different target
The diagnosis-learning experiment is different. On a separately frozen release, full-diagnosis fine-tuning over 1,656 source tasks raises Qwen3-8B exact-step agreement with internal teacher labels from 47.2% to 63.6%, averaged over three seeds on 943 holdout cases. The strongest prompted reference scores 54.7%. Agreement with those labels is not human-verified truth or end-to-end task success. A separate single-seed actor experiment reports a 6.67-point advantage for action-only repair training over success-only training on WebShop-lite. These populations and objectives must not be added together. ↗
Prevent hindsight from entering the input
Mechanism and analysis: preserve what was visible before the decision. If a training example quietly contains the successful reference, a later verifier result or a hidden policy rule, the student may learn hindsight rather than debugging. Group records by source task when splitting: several diagnoses of one failure are related examples, not independent problems. The paper explicitly says the earlier diagnosis release omitted policy system prompts and tool lists and preceded newer screening checks. Its historical results do not validate the revised pipeline or every row in the larger collection. ↗
A disposable test, not a production replay
A concrete test would use a disposable local task with a deterministic verifier. Save the checkpoint, execute an original-action retry and a proposed correction under the same budget, and keep both results, including failures and timeouts. Redact real credentials and private records before storing traces. Do not replay a payment, email or production write merely to see whether a suggested repair works. The original code below checks bookkeeping for three invented replay pairs. It was executed locally; it does not run an agent, estimate significance or reproduce the paper.
Diary analysis: keep explanation and outcome apart
Diary analysis: I would store a diagnosis, its supporting observation and its replay result as different fields. A dashboard that turns an eloquent explanation into a green recovered status loses the most important distinction. A passing continuation is useful evidence under its tested conditions, while the original task may still contain untested requirements. The paper is a preprint, and this article is a source review. I did not download the dataset or train a model. The useful lesson is narrow: failure traces become training evidence only after the objective, available information and comparison are named.
The disposable test, in code
pairs = [
{"task": "a", "checkpoint": "s1", "budget": 10,
"retry": False, "repair": True},
{"task": "b", "checkpoint": "s2", "budget": 10,
"retry": True, "repair": True},
{"task": "c", "checkpoint": "s3", "budget": 10,
"retry": False, "repair": False},
]
assert len({p["task"] for p in pairs}) == len(pairs)
assert all(p["budget"] == 10 for p in pairs)
for field in ("retry", "repair"):
print(field, sum(p[field] for p in pairs), "/", len(pairs))
# Invented inputs: retry 1/3, repair 2/3. Not research results.