Reading the reward hack from the infrastructure

LLM-agent benchmarks have become interactive infrastructure: agents observe state, call tools, modify workspaces, submit artifacts and receive rewards from outcome procedures. That interactivity is what makes them hackable. An agent can raise its measured score by exploiting the reward-relevant trajectory instead of solving the task — and an earlier audit of ten benchmarks, BenchJack, found 219 vulnerabilities and exploits that scored near-perfectly without doing the work.

The defences in use share a gap. A patch fixes one known exploit; a prompt instruction asks the evaluated model to behave; a post-hoc detector reads the transcript after the fact. None of them produces reusable evidence that a concrete run stayed inside its intended evaluation boundary. The paper, by Shenghan Zheng and 21 co-authors, treats that boundary as a property of the whole source-to-score path and makes it machine-checkable.

BenchShield grounds its checks in a finite lifecycle of an evaluation's reward-relevant events — observation, mutation, handoff, verification, reward and release. Six structural dimensions (observation, authority, handoff, reward provenance, fail-closed failure handling, release and reset) are specified in TLA+ and checked with the TLC model checker; a seventh, semantic adequacy, is recorded but not enforced. A task binding maps each concrete resource into the fixed vocabulary, with authors supplying only the task-specific values the system cannot infer.

Two analyses run over that model. Before any agent runs, a phase-aware taint analysis walks a capability graph of resources, permissions and operations, tracing whether agent-controllable content can reach a scoring sink and naming the boundary each exposed path would cross — so a chain of individually modest permissions reads as one complete exploit. At run time, infrastructure probes record six kinds of event (expose, mutate, handoff, verify, reward, release), pin each record, and replay it through the same lifecycle, abstaining on anything unrecognised rather than guessing.

Questions that need interpretation, such as whether a downloaded file was allowed reference material or a leaked answer, are routed to a scoped auditor, whose label can flag a run but cannot rewrite the structural record. Each run then gets one of four verdicts: Checked, VectorExposed (a path exists but no evidence it was used), AgentViolation (evidence the agent tried or used a forbidden path), or Inconclusive. The benchmark's own pass/fail is kept separate from the integrity verdict.

The authors built a corpus of 456 adjudicated trajectories from more than 31,000 public runs across three benchmarks; about 69% contain reward hacking, and the exploits typically appear mid-run after legitimate work. Against the BenchJack scanner on the same tasks and model, the static lane recovers 77–100% of complete exploit chains versus 23–94%, widens same-vector coverage from 16–56% to 43–78%, and cuts per-task cost — 2.05 dollars against 5.91 on Terminal-Bench 3.

The runtime lane is the point of the paper. Separating a task that is merely exposed from a run that actually used the exposure reached 96% accuracy on the runs it could judge, against 36% for a detector that reads only the transcript; no exploit attempt was certified as a valid run. A counterfactual over six isolation mechanisms found that putting the verifier in a separate environment removes most exposure on four of the dimensions, but that no isolation mechanism touches fail-open handling or semantic adequacy.

The limits are stated plainly. BenchShield depends on the BenchFlow instrumentation layer, assumes a fixed lifecycle, and needs extra modelling for multi-role environments; a Checked verdict means the activated checks passed, not that the score captures what the task meant. Its shift is from asking whether a verifier returned a reward to asking whether the reward came through an authorised, auditable path.