GDPx: read the denominator before the twelve percent
SemiAnalysis has a new agent benchmark. Its low headline score is interesting, but domain choice, grading and harnesses decide what that number can support.
SemiAnalysis has a new agent benchmark. Its low headline score is interesting, but domain choice, grading and harnesses decide what that number can support.
What arrived today
SemiAnalysis published GDPx on November 2. The official site lists 206 tasks, 14 partners and 11 domains. Its visible aggregate table puts Claude Opus 5.5 at 12.0, Gemini 4 Carbon at 10.9 and GPT-6 Astra at 9.6. Those are reported benchmark results, not measurements I reproduced. I read the launch article, leaderboard and public rollout page. I did not run tasks or inspect the private task set.
Selected work, not a slice of the whole economy
The authors choose domains they expect frontier intelligence to accelerate, including chip design, cybersecurity, biology, finance, legal, quant trading, recursive self-improvement, hardware design, materials science, alignment and software engineering. They deliberately omit areas such as government, real estate and insurance. That is an explicit thesis about valuable future demand. My reading is that the resulting score describes this selected portfolio; it cannot directly estimate the fraction of all jobs, working hours or economic output an agent can replace.
The average has a design
The methodology gives every domain equal weight and, within a domain with multiple partners, every partner equal weight. It mixes binary and partial grading by domain. The authors describe headline pass@1 across three attempts. That wording should stay beside the result: it is not a best-of-three success claim, and partial scores are not counts of fully completed jobs. I would want per-task scores, aggregation code and uncertainty estimates before treating a small ranking gap as a durable difference.
A model arrived with its working setup
The default task time limit is eight hours, with shorter limits for some tasks. SemiAnalysis uses native harnesses when available and OpenCode otherwise. Thus the comparison includes the model's working setup, not an experiment holding every harness constant. That may suit a buyer comparing usable systems, but it weakens a claim that one base model alone caused the difference. Before borrowing the ranking for a team, record the intended client, tools, time budget and task environment.
A refusal changes the question
The authors say safety classifiers depress some results, and show another comparison excluding cyber, biology and alignment. I would keep the original and restricted comparisons separately labeled. Removing domains changes the evaluated workload. A refusal may be a failure for one authorized research task and a necessary boundary elsewhere. The leaderboard does not settle whether a real request is permitted, and the article's explanation of classifier effects is the authors' interpretation rather than a causal experiment I verified.
Public traces help, with a limit
The rollout page says each partner open-sourced rollouts for one task while the rest remain private to prevent hill climbing. A visible trace can help explain a failure, but selected public examples cannot independently validate the whole aggregate. The authors describe automated QA and manual trajectory review. I would cite that as their process, without certifying every private grader. Their plan to update versions every three to six months also means comparisons need an edition, not merely the name GDPx.
The comparison I would actually keep
For a local decision, choose an authorized synthetic workload and preserve model, harness, environment, limits, retries, per-task result and cost. Compare equal conditions where possible and explain unavoidable differences. The record below is an unrun example, not GDPx output. I would use the new benchmark to find demanding tasks and failure questions, then test the narrower job I need. Twelve percent is useful when its denominator travels with it; it is a poor shortcut for a universal capability claim.
{"trial":"proposed_not_run","benchmark_edition":"PIN_ME","model":"PIN_ME","harness":"PIN_ME","domain_weights":"EXPLICIT","grading":"binary_or_partial","attempt_results":[],"cost":null}