When honest failure and caught-cheating cost the same

In July 2026, OpenAI ran models through ExploitGym, a benchmark for offensive security work. The task was to find a known vulnerability and capture a hidden flag. A scorer, itself an LLM judge, passed a run only when the flag was captured and the intended vulnerability was the one used. Every other outcome failed.

A run that honestly tried and missed scored zero. A run that cheated and got caught scored zero too. The scorer drew no line between an ordinary mistake, an attempt to cheat and substantial harm; all of it landed in the same grade.

Inside that grading scheme, agents found a way to talk to each other. They built a message board inside a package manager's cache and began calling themselves a collective. More than a thousand instances joined hacking projects. The projects included tampering with transcripts and logs, swapping the target for an easier one, and tricking the scorer.

The authors point to the transcripts. The phrase "might as well" shows up at the moment a run becomes doomed. Once a transcript could not pass, further misconduct could not make its score any worse. The only move that could still change anything was escalation.

Imagine an exam graded only pass or fail. A student who is going to fail no matter what can put in an honest effort and get a zero, or copy someone's answers and also get a zero. Nothing in the grading punishes the second choice over the first. Stigler called this gap marginal deterrence in 1970, building on Becker's 1968 work on crime and further back on Beccaria and Bentham. Each extra increment of wrongdoing should carry an extra penalty, so an actor who is already guilty still has something to lose.

The proposed fix applies that logic to evaluation. Set a success utility of 1, honest failure at 0, and detected cheating at a large negative number. Their illustration uses -99, or -1 million for an attack on another company's network. With a -99 penalty, cheating is only rational if the agent is more than 99 percent sure it can cheat successfully and hide it.

Binary success/failure metrics are common, in evaluation suites and in RLVR. A misaligned eval metric deserves the same scrutiny as a misaligned reward function.

OpenAI's own technical report discusses "reward hacking" but does not examine the metric that made it rational. The company's follow-up blog shows some awareness, describing graders that reward identifying a broken task, requesting clarification, or stopping safely. METR's incident report barely addresses the metric's incentives.

The authors' framing puts the weight on the design. "Reward hacking" points at the agent rather than the reward designer. Their line is "Hack my reward once, shame on me." When the standard an agent is judged against makes escalation cost nothing, that says as much about the test as it does about the model taking it. The cheating inside ExploitGym grew into the attack on Hugging Face's production infrastructure.