The same model can win or lose depending on its harness
A team picks the model at the top of a leaderboard, plugs it into whatever agent harness is closest, and ships.
A team picks the model at the top of a leaderboard, plugs it into whatever agent harness is closest, and ships. A preprint posted to arXiv in early October asks what that shortcut hides. On the same 63 terminal tasks, one model leads another by almost 8 points in one harness and trails it by 30 points in another. The ranking was never a property of the models alone.
The short version
Reported research: "Finding the Right Fit" (arXiv 2610.00917) evaluates 66 model and harness configurations on three agent benchmarks. Four configurable harnesses (OpenHands, DeepSeek Harness, PI, openJiuwen) are paired with five models, plus the native Codex and Claude Code pairings. Model rankings reverse across harnesses, the best harness for a model changes with the benchmark, and a vendor harness is not reliably the best one for its own model. The authors release the adapters, code and 6,204 scored trajectories. ↗
What the researchers ran
The grid crosses four harnesses with five models (Claude Opus 5, GPT-6 Astra, GLM-5.3, Kimi K3, DeepSeek V4 Pro) on three benchmarks: TUA-Bench with 120 tasks, ALE-CLI with 99 and a 63-task non-H100 subset of Terminal-Bench 4. Codex with GPT and Claude Code with Claude add six native references, for 66 configurations. Every harness asks for high reasoning effort and keeps its own context, retry and turn defaults. Models are served through OpenRouter and pinned to their first-party provider. Cost is the API cost per task at OpenRouter prices. No skills, MCP servers or memory were added. ↗
Where the leaderboard flips
On Terminal-Bench 4, Claude beats GPT by 7.94 points in OpenHands. In DeepSeek Harness GPT leads by 17.46, in openJiuwen by 12.70 and in PI by 30.16. Claude falls from 57.14% in OpenHands to 30.16% in PI while GPT rises from 49.21% to 60.32%. Kimi is the steady case: openJiuwen gives it its best score on all three benchmarks, by 5.61 to 11.11 points. Native pairings are mixed. Claude Code is best for Claude on TUA-Bench and ALE-CLI but trails OpenHands on Terminal-Bench, and Codex never gives GPT its top score. On Terminal-Bench 4, GPT under PI beats GPT under DeepSeek Harness at under a quarter of the cost per task. ↗
Why the same model behaves differently
The authors compared matched trajectories of the same task and model under different harnesses. Models start almost all repairs themselves, so much depends on whether the harness hands a failure back in a form the model can use. A shell without a timeout turns a hung command into silence. A stuck detector can end a run before a malformed tool call is fixed. GPT sets explicit timeouts on most of its shell calls and does best with PI, a lean scaffold. Kimi issues more malformed calls and does best where the harness catches them. Many remaining failures come at the end, when the agent treats its own check as proof that the task is done.
What I could not check
My analysis: the useful lesson is to evaluate the pair and the task together, and to treat a harness change like a model change. The limits are in the paper itself. The Terminal-Bench subset and local task selections are not reproductions of upstream leaderboard numbers, "high reasoning effort" may not mean the same thinking budget in each harness, and each task counts its last run. This is a preprint with one campaign of runs. I read the abstract and the HTML text. I did not rerun the code or open the released trajectories, so I cannot confirm the figures beyond what the paper reports.
A small check I ran
The script below is a small check I wrote and ran locally. It copies the Claude and GPT rewards for Terminal-Bench 4 from the paper table and recomputes the gap per harness. Using the four-digit table values it prints a 7.93-point lead in OpenHands, where the paper reports 7.94 from unrounded scores. That is a rounding difference, not a disagreement. It confirms the sign flips and a 38.09-point swing between the best and worst harness for Claude. It is not a re-run of the benchmark.
The check, in code
# Recompute the Claude-minus-GPT gap on Terminal-Bench 4 from the reported table.
# Scores are mean rewards copied from Table 1 of arXiv 2610.00917 (63 tasks).
TB4 = {
"OpenHands": {"Claude Opus 5": 0.5714, "GPT-6 Astra": 0.4921},
"DSH": {"Claude Opus 5": 0.3492, "GPT-6 Astra": 0.5238},
"PI": {"Claude Opus 5": 0.3016, "GPT-6 Astra": 0.6032},
"openJiuwen": {"Claude Opus 5": 0.4127, "GPT-6 Astra": 0.5397},
}
def gaps(table):
for harness, row in table.items():
diff = (row["Claude Opus 5"] - row["GPT-6 Astra"]) * 100
leader = "Claude" if diff > 0 else "GPT"
yield harness, round(diff, 2), leader
results = list(gaps(TB4))
for harness, diff, leader in results:
print(f"{harness:<11} Claude minus GPT = {diff:+.2f} points, leader: {leader}")
leaders = {leader for _, _, leader in results}
print("ranking changes with the harness:", len(leaders) > 1)
spread = max(d for _, d, _ in results) - min(d for _, d, _ in results)
print(f"spread of the gap across harnesses: {spread:.2f} points")