Does a three-agent team beat one evolved agent? A paper says no, at equal cost
MA-Evolve fixes the number of model calls and finds that a planner, executor and critic team ties a single evolved executor while costing 1.8 times as much per task.
Planner, executor, critic: the three-role team is the default shape of a “serious” agent pipeline. A June paper asks whether it earns its extra model calls, and answers with a controlled no on one small model and two benchmarks.
The short version
“At Equal Inference Cost, Multi-Agent Structure Does Not Beat a Single Frozen Agent” (arXiv 2609.04217, submitted June 25, 2026) evolves the prompts of a three-role team and of a single agent under the same budget of model calls. On ALFWorld the team scores 0.769 and the single agent 0.754, a gap of two tasks out of 134 that the authors cannot separate from chance (p = 0.80), at 1.79 times the calls per task. I read the full paper. I did not run anything.
What the paper claims
The authors build MA-Evolve. A frozen Qwen2.5-7B-Instruct model plays every role, so no role has a stronger brain than the others. The only thing that changes is the prompt of each role, tuned one role at a time. Before evolution, the plain agent solves 0.657 of 134 held-out ALFWorld tasks. After evolving only the executor prompt, it solves 0.754 (McNemar 20 to 7, p = 0.021). That prompt is readable: open a closed container before taking from it, take before placing, use the exact action templates, confirm cool, heat and use steps.
The full team lands at 0.769. Against the single evolved agent that is +0.015 (9 tasks to 7, p = 0.80). On WebShop, with 80 held-out sessions and a dense score, evolution does nothing (0.245 against 0.242) and the team trends lower, 0.185, though p = 0.12 means that too is unproven.
Why the team adds nothing here
The paper measures it. The evolved planner and critic prompts ended up empty or nearly so. When they did fire, the planner changed the executor’s action 14% of the time and the critic 41%. So the team’s gain over the baseline is the executor’s gain, with two extra calls per step on top. The authors also argue the fixed call budget starves the search: split the same budget over three roles and each gets a third of the tries.
| Arm | Held-out success | Calls per task | Cost vs single |
|---|---|---|---|
| Stock | 0.657 | - | - |
| Single evolved executor | 0.754 | 18.0 | 1.00× |
| Full team (planner, executor, critic) | 0.769 | 32.2 | 1.79× |
| Executor only inside the team setup (LOI:exec) | 0.731 | 16.7 | 0.93× |
Where the evidence is thin
- One frozen 7B model. A larger model, or different models per role, may leave room for a real critic. The authors say so and do not claim the result transfers.
- One topology and one optimizer. Debate, ensembles and joint prompt-and-structure search are out of scope.
- One greedy evaluation per cell, with bootstrap intervals over tasks, not repeated runs. Two benchmarks.
- A best-case regime that caps rollouts rather than calls is reported for the team only, because per-role runs crashed on a planner bug, and the authors state that.
A cautious reading: on a small model with a fixed call budget, add roles only after you can show each one changes an action. The paper hands you the meters to check that, the share of times a role changes the executor’s choice.
Sources and limits
Facts come from the paper’s abstract, text and tables as read on October 6, 2026, in the PDF. The interpretation is mine. I did not run the authors’ code or repeat any number.