LiteTrajEval: cheaper diagnosis needs honest denominators

Reported design: one online judge, offline preparation

The preprint "Lightweight, Rubric-Guided Trajectory Evaluation for Production AI Agents" builds domain-specific rule profiles offline from representative outputs. Online, it normalizes a trajectory, marks suspected failure signals, compacts repeated material and sends one bounded representation to an LLM judge. The paper uses a 50,000-character trajectory budget. Characters are not tokens, and one online judge call does not mean the offline rule-making costs nothing.

The serializer reserves task and final-answer space, preserves step order, gives evidence different retention priorities and shrinks lower-priority long outputs first. Within a step it selects relevant evidence while penalizing redundancy. A final hard truncation is still possible if the budget cannot otherwise be met. This is structured evidence selection, not a guarantee that every causal clue survives.

Read the denominators before the headline

The evaluation uses 44 Magentic-One-style failed trajectories and 29 tau-retail failed trajectories from AgentRx. Every sample is already a failure. Detection Rate therefore measures recall of known failures, not the ability to separate good and bad runs. There is no successful-run set here from which to estimate false alarms.

Alignment is conditional on detection and compares predicted steps with human-annotated failure groups. At @3, a prediction within three steps of a reference region can count. With GPT-5-mini, Magentic-One detection is 43/44 for both AgentRx and LiteTrajEval; conditional @3 alignment rises from 35/77 to 64/77. On retail, detection changes from 25/29 to 26/29 and conditional alignment from 16/27 to 23/28. These are different denominators, not overall accuracy percentages.

Cost savings are measured on this batch

With the same GPT-5-mini judge, the paper estimates total evaluation cost at $2.4628 for AgentRx and $0.4155 for LiteTrajEval across the 73 traces, about six times lower. It reports total runtime of 15,990 versus 1,840 seconds, about 8.7 times lower. The larger reported cost reduction with DeepSeek-v4-flash also changes the judge, so it is not the same-model comparison.

The production pilot is described qualitatively: an internal SRE agent, with plausible localized failures in preliminary manual inspection. That traffic has no step-level ground truth. It cannot establish the public-test alignment rate in production. I read the paper; I did not reproduce these figures or inspect the private deployment.

My analysis: keep a compression audit beside the diagnosis

Before using a short trace to decide a fix, save the raw trace privately and assign every retained excerpt a step ID and source offsets. Record which sections were removed and why. Compare the diagnosis on the compact version with a human review of the full evidence on a small labeled set. Include successful runs as well as failures so false alarms become visible.

An illustrative cancellation case shows the risk: the final tool response may say "success" while an earlier user constraint made that cancellation wrong. Keep the constraint, selected target and action result together, even if the success response is short. That example is a proposed check, not a new measured result. Rule profiles need fresh review when output formats or domains change.

Limits and a useful decision

A cheaper diagnosis is valuable only if it still catches the failures you care about. This paper offers evidence for conditional localization on two small failed-trace datasets, not a universal production detector. I would evaluate detection coverage, false alarms, localization tolerance and cost separately. A single score hides the part of the system that chose not to flag a trace.