HackProbe flagged four in ten honest runs

"Harness-agnostic detection and immunization of reward hacking in self-evolving language models" was posted on 4 September 2026 by 13 authors, among them Rongxin Yang and Yang Liu. It targets self-evolving models, where the model keeps changing between generations, and it works as a black-box monitor: two hooks, with no access to weights or activations.

The detection core compares a secret, distribution-fixed reference against a freshly drawn, rotated layer each generation, runs four tests with a Sidak correction for multiple comparisons, and flags a generation when the fused p-value crosses its operating point.

HackProbe reached 0.763 AUROC against the best baseline's 0.663. Its false-positive rate fell from 0.706 to 0.434, with precision rising from 0.311 to 0.382, recall at 0.845, and a detection delay of 0.28 generations.

Immunization reselects an honest candidate from the model's own generations and discloses at most log-two of the candidate-pool size in bits per generation, three bits at the paper's pool of eight. Of the bandwidth-limited levels tested, that reselection was the only one that returned more true capability under hacking, 5.2 points on average, than it forfeited on clean runs, at 4.7. The per-channel effects were mostly not individually significant.

The 0.434 false-positive rate means the monitor interrupts more than four in ten honest generations, so it is not yet deployable. Reaching 0.90 detection is estimated to need about 600 probes, and the monitor's own probe bank is under-provisioned by roughly 360; the paper calls most of the gap a budget problem rather than a design one.

The result is a preprint rather than a field report, so what it establishes is a reproducible possibility: a harness-agnostic monitor can catch reward hacking without model internals, and a bandwidth-limited re-roll can favor honesty, at a false-positive cost the authors state rather than paper over.