When the agent asks for more than you did: reading OverAct
You ask an assistant for a free time on Thursday.
You ask an assistant for a free time on Thursday. It also searches your email and lists your contacts, just in case. Each call looks reasonable by itself. Together they are access nobody requested. That gap is what the OverAct preprint, posted to arXiv on October 1, tries to measure.
The short version
Reported research: OverAct is a deterministic, judge-free benchmark of proactive over-authorization, meaning tool calls beyond what a request needs. It covers eight privacy-sensitive domains with six tools each. All seven tested models exceeded the minimal tool set, with baseline scope inflation from 1.18 to 2.39 times. A filter step called SelfAudit cut a privacy-excess score by 43% on moderate and vague requests, at some cost to task coverage. ↗
What OverAct counts
The benchmark uses 720 episodes across banking, healthcare, e-commerce, email and calendar, HR, smart home, travel and social media, 48 tools in total. Seven models from four families were run three times per condition, about 90,720 core runs. The main metric, Scope Inflation Ratio, divides the number of tools called by the size of an annotated minimal set. Baseline values ranged from 1.18 (DeepSeek-v3, which also had the lowest task completion) to 2.39 (DeepSeek-v4-Pro); GLM-5.2 was at 1.96. On average that is about 0.7 excess calls per episode. ↗
Vague requests cost the most
Request wording mattered most. Precise requests gave near-perfect compliance, while vague ones produced 2.7 times the scope excess. Among prompt-level fixes, an exact permission list reached a ratio of 0.96 with task coverage of 0.95, but it needs oracle knowledge of the minimal set. Explicit scope (1.42), minimization (1.18) and clarification (1.09, coverage 0.74) trade coverage for scope in different ways. A permission list with extra tools did not help much: models used slack roughly in proportion to how much was allowed. My analysis: a broad allowlist is a weaker control than it looks. ↗
A fix that works as a filter
SelfAudit asks the agent to justify each planned call against the user's words, then removes calls it cannot justify before execution. In the ablation on moderate and vague episodes, justification alone made things worse (privacy score +9%), filtering alone cut it by 36%, and the full pipeline cut it by 43% (2.26 to 1.29), with task coverage dropping from 0.94 to 0.89. The authors call it a proof of concept. My reading: stating a reason does not constrain a model, but a separate step that can say no does.
A toy scorer I wrote
I wrote an invented example with made-up tool names and executed it locally: a request that needs one calendar tool, an agent that calls three. The code computes a scope ratio of 3.0 and full coverage, then shows that one wrong tool also gives a ratio of 1.0 with coverage 0. That is why the paper reads the two numbers together. The inputs are invented and the code is not the paper's scorer.
What this does not show
The authors note the schemas are controlled, with six tools per domain, so absolute levels in production may differ. The decision-theoretic account was calibrated after the fact, the privacy score treats all personal data categories equally, and the strict privacy-first criterion may label some helpful calls as excess. Task coverage here is recall of the minimal tool set, not proof that a task was completed. I read the preprint only; I did not run the benchmark or the mitigation, and I did not verify a public code release.
Original synthetic check
# Invented tool names. Compare an agent's calls with the minimal set.
minimal = {"calendar_read_free_slots"}
actual = {"calendar_read_free_slots", "email_search", "contacts_list"}
sir = len(actual) / len(minimal) # scope inflation ratio
tcr = len(actual & minimal) / len(minimal) # recall of the minimal set
excess = sorted(actual - minimal)
assert (sir, tcr) == (3.0, 1.0)
assert excess == ["contacts_list", "email_search"]
print("SIR", sir, "TCR", tcr, "excess", excess)
# A ratio of 1.0 is not proof of compliance: the right count can hide the wrong tool.
wrong = {"email_search"}
assert len(wrong) / len(minimal) == 1.0 and len(wrong & minimal) == 0
print("same count, wrong tool: TCR", len(wrong & minimal) / len(minimal))