ExplorationBench: when an agent has to discover the rules

RESEARCH DIARY · 27 SEP 2026

The manual says positions start at zero. The tool, asked for position one, hands back the second character, which fits the manual. Then it does something the manual never mentioned. Should the agent trust the document, throw out the tool, or design a cleaner test? ExplorationBench turns that dilemma into a score, using two executable worlds whose hidden rules deliberately contradict the ones we know. It is a September 24 preprint about software worlds, not about real laboratories.

The protocol

AlienCode is a tiny programming language with 31 discovery targets; AlienLogic is a proof system with 24. Each has 70 held-out tasks. A system gets a flawed manual, fixed worked examples and a tool-call schema. It chooses probes for four rounds and reads executable feedback. At milestones it has no tools: it reports its inferred rules and answers the held-out tasks. Ten systems each run three independent trajectories per world. The authors rank a system by its best final trajectory, Best@3, which differs from typical-run performance.

A discriminating probe

In AlienCode, integer literals are XOR-ed with 27, so EMIT(100) prints 127, and PLUCK positions begin at one despite the manual. A useful test changes one feature at a time: compare EMIT(100), EMIT(101), then PLUCK at adjacent positions. A single surprising output does not identify which rule changed. A probe should distinguish rival hypotheses, not just collect more tokens.

What the results say

The authors report a best AlienCode trajectory of 87.6% after four rounds; matched turns without environment feedback give 0.5–11.0%. Giving a system its own best probes back as a fixed sequence lowers accuracy for 9 of 10 systems in AlienCode. Two off-by-one rules enter 51 of 70 tasks, yet questions whose needed rules were correctly stated were solved only 70.9% of the time. In AlienLogic, supplying the full rules yields 93–97%, above autonomous exploration. These figures are the authors’ tests, not independent replications.

Why one score misleads

Final trajectories for the same system and budget can differ by up to 72.8 percentage points. Six of 30 AlienCode trajectories finish at least three points below an earlier milestone. Best@3 therefore answers “can one of three runs succeed?” rather than “what happens on the next run?” Report the full distribution, baseline before probing and changes at every milestone. The two worlds do not rank systems alike: the paper reports a Spearman correlation of 0.35.

Analysis and practical test

My reading: this benchmark separates evidence selection from rule application better than a static quiz does. In a local sandbox, hide one changed parser rule, give an agent only a deliberately outdated spec, log each query and score ten unseen inputs after each round. Repeat with different seeds; compare the chosen probes against a fixed sequence and a no-feedback condition. Do not let the exploratory agent edit the scorer or the hidden rule file. The sample gate below illustrates measurement, not the paper’s implementation.

Limits

Both worlds are deterministic synthetic constructions. An exact interpreter or proof checker grades them; real scientific claims often have noisy measurements, changing conditions and uncertain ground truth. The paper does not show a system discovering a real law. A successful rule report is not the same as a reliable answer or a safe action.

Illustrative measurement pseudocode, not study code

# Explanatory pseudocode: do not expose held-out answers
for step in range(4):
    observation = world.run(agent.choose_probe(history))
    history.append(observation)
    score[step] = grader.check(agent.answer(test_tasks, tools=False))
report(score, all_trajectories=True)

Source and status: ExplorationBench, arXiv:2609.30199 · full text. 24 Sep 2026, preprint; author-reported results.