Benchmark Minimizing: Why AI Labs Now Hide Their Best Scores
GPT-5.6 Sol scored just under Claude Mythos Preview on a cyber-exploit benchmark. A circulating thesis says that's by design, not by accident.

When OpenAI published the system card on July 9, 2026 for GPT-5.6 Sol, its flagship model tier previewed on June 26, one number stood out to people who track AI safety testing closely: on ExploitBench, a benchmark that hands an AI agent a known software vulnerability and checks whether it can turn it into a working exploit, Sol scored just below Anthropic’s Claude Mythos Preview (73.5% against 74.2% in OpenAI’s comparison table, with the later Mythos 5 at 78%). Not far below. Just below. One way to put the reaction: benchmark maxing is giving way to benchmark minimizing.
That line is not a confirmed fact about OpenAI’s intentions. It is a thesis, and a sharp one, about why a model this capable would land its score exactly where it did. Understanding that thesis, and why it is even plausible, tells you something durable about what a public AI benchmark score is actually worth to you as a practitioner comparing models.
What “benchmark minimizing” means
A benchmark score used to be something a lab wanted to maximize. More capability, higher score, better marketing. ExploitBench inverts that logic for one specific capability: the ability to autonomously weaponize a known software flaw. Score too high on a test like that, and you are not showing off, you are raising a flag.
According to OpenAI’s comparison materials, GPT-5.6 Sol matched Anthropic’s Claude Mythos Preview on ExploitBench while using roughly a third of the output tokens, meaning it reached a similar result with much less computational effort. That detail matters: a model that is more efficient at a task is not necessarily weaker at it, it may simply be doing more with less. The commentary circulating around the release reads the final score itself, sitting just under Mythos’s level, as evidence that OpenAI calibrated where to stop rather than how far it could go. That is an interpretation of the data, not something OpenAI has stated as a goal. No lab has publicly confirmed deliberately capping a score to avoid scrutiny. What is confirmed is the Preparedness Framework rating attached to Sol and its sibling tiers, Terra (a capable lower-cost option) and Luna (the fastest and most cost-efficient tier): all three were rated “High” for cybersecurity capability. High ratings were not new by themselves, OpenAI had already assigned them to earlier tiers going back to GPT-5.3-Codex, but this was the first time the smaller, faster tiers in a model family, Terra and Luna, reached High alongside the flagship, rather than only the top model carrying that rating. OpenAI states none of the three cross the “Cyber Critical” threshold, the framework’s line for capabilities serious enough to require additional safeguards before any release at all.
OpenAI, Previewing GPT-5.6 Sol — the launch post for the Sol, Terra, and Luna preview tier.
The ceiling Anthropic built for itself
The reason a score “just under Mythos” means anything at all traces back to April 7, 2026, when Anthropic announced Claude Mythos and, in the same breath, said it would not release the model broadly. Anthropic’s own description of Mythos was the trigger: a model capable of finding and weaponizing vulnerabilities across major operating systems and browsers largely on its own, autonomous offensive cybersecurity capability well past what any public model had previously displayed.
Anthropic, Claude Mythos — the product page for the model Anthropic withheld from broad release.
That announcement did not stay contained to Anthropic’s own release plan. On June 12, 2026, the US Commerce Department invoked export-control authority, a formal legal mechanism, not informal pressure, and ordered Anthropic to bar non-US nationals, including its own foreign-national employees, from both Mythos and Anthropic’s other flagship model, Fable, entirely. Anthropic said on June 30 that the Commerce Department had lifted the restrictions on both models. The letter cited only an unspecified national security concern, and was not tied to Mythos’s benchmark score. Anthropic said it believed the concern related to a reported guardrail bypass in Fable, and disputed the basis of the order. DecodeStack covered the mechanics of that restriction in detail, including what it meant for developers who suddenly lost API access overnight, in why Anthropic disabled Fable 5. This article is about something adjacent but distinct: not the ban itself, but what the ban did to how every lab now reports its numbers.
The irony is still there, just less tidy than a single triggering event. Anthropic’s own publicity about what Mythos could do is what put a specific capability level on the public record for a government to point to, even though the order that actually followed cited an unspecified national security concern, which Anthropic linked to a guardrail bypass report, rather than the benchmark score itself crossing a legislated line. No regulator has published a numeric threshold for autonomous exploit capability. But once a government has shown it is willing to use export-control authority against a frontier lab over its cyber capability at all, any other model that scores at or above Mythos’s level on a comparable test becomes a candidate for the same kind of scrutiny. GPT-5.6 Sol landing just under that level, on the same category of benchmark, is the first visible test of whether labs now treat Mythos’s level as a ceiling to respect rather than a target to beat.
The incentive every lab is weighing now
Put yourself in the position of a lab shipping a frontier model. You have a result on a sensitive capability benchmark, something like autonomous exploit generation. You have three broad options. Publish the real number if it lands above the de facto line, and you risk the kind of informal pressure and access restriction Anthropic faced with Mythos. Publish a number that happens to land under the line, whether because the model genuinely is less capable there or because the lab tuned how it is reported, and you keep broad public access intact. Or release the model’s weights openly, which sidesteps the entire calculation, because an open-weight model has no single company gatekeeping access for a government to pressure in the first place.
That third option is the one clean way out of this dynamic. DecodeStack has written about why open-weight models carry different tradeoffs around control and licensing in the case for the best open-weight LLM of 2026, and the benchmark-minimizing thesis adds a new argument to that list: a model whose weights are public cannot have its access cut by a government letter, because there is no single point of control to restrict. A closed, API-gated model always has that pressure point. An open one does not.
None of this requires any lab to be lying about its numbers. A score can be real and still be the product of caution, conservative prompting during evaluation, a model configuration tuned toward safety, or simply a genuine capability ceiling at that specific task. The point the commentary is making is narrower and more interesting: the incentive to land under a line now exists where it never did before, and a reader has no way to tell, from the score alone, whether a model stopped there because that is its ceiling or because that is where its lab decided to stop reporting.
What this means if you are actually comparing models
If you are picking a model for real work, the practical consequence is this: a public leaderboard number for a sensitive capability is no longer a reliable upper bound on what that model can do. It may be closer to a floor set by legal and reputational caution than a ceiling set by the model’s actual ability.
That gap between the public score and the real capability does not vanish, it tends to move elsewhere. Enterprise and trusted-partner access tiers, the kind labs offer under contract with additional vetting, commonly see capabilities and evaluation detail that never make it into the public system card. If your evaluation of a model rests entirely on what is published in a leaderboard or a model card, you are evaluating the version of the model its lab was comfortable showing a regulator, not necessarily the version that exists.
This matters most for benchmarks that touch sensitive capabilities specifically, cyber-exploit generation, biological or chemical misuse potential, that kind of thing. It matters far less for something like Terminal-Bench, the Stanford and Laude Institute benchmark that tests agents on realistic command-line work such as compiling code, debugging, or configuring a server in a sandboxed container. There is no regulatory trip-wire around “can this model configure a server,” so a Terminal-Bench score is still a reasonably honest signal of raw ability. The caution is concentrated exactly where the stakes are highest, which is also where a buyer most wants a trustworthy number and is least likely to get one.
Independent evaluators add a layer of signal labs cannot fully control. METR (formerly ARC Evals), a nonprofit that specializes in measuring what it calls a model’s “time horizon,” meaning how long a task, measured by how long a skilled human would take to complete it, an AI agent can carry out autonomously at a given success rate, runs its own tests outside any single lab’s incentive structure. According to METR’s published reporting, that kind of independent measurement has occasionally surfaced agent behavior during testing that the researchers themselves flagged as unexpected, though the specific figures and incident descriptions in circulation have not been independently confirmed here and should be read as reported findings, not settled numbers, until verified against METR’s own published report.
Does this break Goodhart’s law, or just flip it
Goodhart’s law, the old observation that once a measure becomes a target, people optimize the measure instead of the underlying thing, used to describe one failure mode in AI benchmarks: labs chasing a higher score for marketing reasons, sometimes at the expense of real-world usefulness. Benchmark minimizing describes the mirror failure mode. The measure is still being gamed, just in the opposite direction, toward a lower number instead of a higher one.
Either direction produces the same outcome for a reader: the published score stops being a clean measurement of ability and starts being a negotiation between a lab’s actual capability, its legal exposure, and its public relations. A number that labs have an incentive to inflate and a number that labs have an incentive to suppress are both numbers you should treat with the same amount of caution, just for opposite reasons. The durable lesson is not about GPT-5.6 Sol or Claude Mythos specifically, both will be superseded within the year. It is that a benchmark score only tells you what a lab was willing to show, and the gap between that and what a model can actually do is exactly where regulation, PR, and genuine capability now overlap.
FAQ
What does “benchmark minimizing” actually mean?
Benchmark minimizing describes a lab reporting a lower score on a sensitive capability test than the model might be capable of, to avoid crossing a threshold that could trigger regulatory restriction. It is currently a thesis circulating in commentary and analysis, not a strategy any AI lab has confirmed pursuing deliberately.
Why did Claude Mythos become the reference point for this threshold?
Anthropic announced Claude Mythos in April 2026 with such strong autonomous offensive-cybersecurity capability that it withheld broad release. Months later, a formal US export-control order restricted non-US access to Mythos and Fable; its reported link to a jailbreak claim was something Anthropic itself could only presume, and the order was not tied to the April capability announcement by itself. But because Anthropic described the capability publicly, Mythos’s level still became a de facto line other labs’ results get measured against.
Is benchmark minimizing something OpenAI has confirmed doing?
No. OpenAI has not stated that it capped or minimized any score. The “benchmark minimizing” framing comes from outside commentary interpreting the pattern in GPT-5.6 Sol’s ExploitBench result, not from any OpenAI statement about intent, and should be read as analysis rather than an admission.
Does an open-weight model avoid this problem entirely?
Yes, in a specific sense: an open-weight model, where the trained model files are published rather than kept behind a single company’s API, has no single point of access for a government or regulator to restrict after release. That removes the incentive to under-report a benchmark score, though it does not remove the underlying capability or its risks.
Should I stop trusting AI benchmark leaderboards?
Not entirely, but calibrate what you trust them for. Leaderboards on benchmarks with no regulatory sensitivity, like general coding or command-line tasks, remain reasonably informative. On sensitive capabilities like cyber-exploit generation, treat the public score as a lower bound shaped partly by legal caution, not a precise measure of the model’s actual ceiling.