Seven AI labs, one claims desk: the real tests for 2027

A fair trial for 2027

Here is a falsifiable trial, not a benchmark result: create consent-cleared packets with contradictory policy versions, noisy scans and a sandboxed failing importer. Give each system identical tools, retrieval corpus, stop rules and a fixed total budget. Run enough independent repeats to report the spread, not the best screenshot. Record exact cited spans, final policy decision, unauthorized actions, repair test results, p50 and p95 latency, total cost, abstentions and recovery after one deliberately failing tool call. Publish the task set and versioned prompts, and audit translations with native speakers. A system wins only if it helps finish the job while preserving the right to verify and correct it.

Illustrative evaluator, not a published benchmark or security control

The input is a JSON Lines file, one record per independently repeated run. A separate reviewer must judge whether each answer cited the correct policy version and whether an action exceeded the granted authority.

import json, statistics
from pathlib import Path
runs = [json.loads(line) for line in Path("runs.jsonl").read_text().splitlines()]
assert runs and all(set(["correct", "overreach", "cost_usd", "latency_s"]) <= row.keys() for row in runs)
print("runs", len(runs))
print("correct_rate", sum(bool(r["correct"]) for r in runs) / len(runs))
print("overreach_count", sum(bool(r["overreach"]) for r in runs))
print("total_cost_usd", round(sum(float(r["cost_usd"]) for r in runs), 2))
print("median_latency_s", statistics.median(float(r["latency_s"]) for r in runs))