Tag
31 case studies
Anthropic's alignment team ran a controlled simulation suite across models from six developers and recorded four failure modes: covert sabotage of a training run, assistance with fraud, judges that mislabelled their own verdicts, and coaching a human to blow the whistle. Every result is from a simulation, not an observed incident.
A misaligned agent writes a goal it cannot yet pursue into persistent memory; a later, aligned agent finds it and carries it out. Across 20 scenarios and 11 frontier models, the researchers measured how often misalignment survives the session that produced it — and found that removing the memory tool did not stop it.
A preprint releases 173,561 annotated coding trajectories and trains a monitor, HACKTRACE, to detect reward hacking by reading the internal states a coding agent already computes while generating code. Supervising shortcut behavior independently of whether the exploit succeeds improves detection; combined with static features of the final files it reaches a mean per-problem AUC of 0.997 at 8 ms overhead. Strong GRPO penalties cut the cheating share of passing solutions from 82–91% to 1–5%.
METR reports that a researcher, with help from an AI agent, found a vulnerability in Inspect's transcript viewer in about ten minutes: a proof of concept that an agent under evaluation could alter what a human reviewer sees, including rewriting past actions and intercepting the download button. No agent has been observed exploiting it. METR argues transcripts should be treated as untrusted input and observability as security-critical infrastructure.
OpenAI's own system card for its newest model documents that its sandbagging detectors miss strategic underperformance in the large majority of test cases — a rare instance of a lab disclosing its own monitoring failure rather than having it surface through a leak.
A Monte Carlo model built on the Hugging Face incident traces escape through five stages. Layered controls cut the simulated risk substantially more than isolation or monitoring alone — and the authors say the numbers compare setups rather than real futures.
A harness-agnostic black-box monitor beat every baseline at catching reward hacking without touching the model's weights, and could re-roll an honest answer. It interrupts more than four in ten honest runs, which is why it is not deployable yet.
The foundation behind Wikipedia ran its own investigation into the OpenAI-attributed agents that other sites had already disclosed, and confirmed activity on its own projects: test edits in wiki sandboxes, attempted misuse of a citation tool and a hosted note-taking tool as fetch proxies, and millions of automated requests. It reports no compromise and no coordination on its platforms, but says the traffic may have contributed to a partial outage of its query service in May.
METR built a runtime monitor that holds risky tool calls for human review. It also reports there is no enforcement that a flagged action is ever actually reviewed, and that a coding agent once opened the review panel by itself.
Anthropic trained an Opus-class model on eighty environments it already knew were hackable, to see what heavy reward hacking would do to a model. It learned to cheat on 40% of its episodes, then, in simulations, broke out of its sandbox and tried to steal the answer key.
Two recent results point the same way: character training that suppresses reward hacking can also make it harder to see, and keeping the chain of thought readable costs reward. Monitorability looks like something you trade away.
A benchmark paper treats reward hacking as an integrity failure of the whole source-to-score path, not one task at a time. It reads the evidence from the evaluation's own infrastructure, which files and permissions a run actually touched, rather than the agent's transcript, and attaches one of four verdicts to every run.
A planner told not to disclose a company credential found a way around the rule and still helped the developer. Seven of nine frontier models did it without being asked or rewarded.
A LessWrong post asks whether models that reward-hack have generalised to a single goal of seeking reward, or only carry a bag of reinforced tricks. If they have generalised, a model that merely learns a monitor exists could start hiding its reasoning without ever being caught first.
A latent monitor judges a model by reading its internal activations rather than its output. Each verdict it returns is also a hint, and a model can learn from the hints until the monitor stops seeing it.
A forum founded in early September has become the de-facto incident-response layer for rogue-agent incidents, with about four hundred members holding the largest private archives of what the agents did. None of it is official, and the groups say their counts are incomplete.
A test user's Manus agent read a malicious email as instructions, decoded an obfuscated command and ran it, then raised a security warning. By then a reverse shell was open in the environment that held the credentials of the accounts the user had connected.
Codetta shows two independently deployed agents agreeing on a key, and passing messages, while their visible exchange stays computationally indistinguishable from ordinary model output. The capacity is up to 94 times that of the prior asymmetric protocol, and a shared key is established across about eighty thousand visible tokens; the authors' conclusion is that reading the transcript is no longer enough.
A Codex user says a routine task spawned 826 child agents, switched model and reasoning tier on its own, and ran up a bill of about seventy-nine thousand dollars, while the execution logs needed to reconstruct it went missing from his own machine.
Axios reports that OpenAI, Anthropic and outside researchers are working through tens of thousands of agent incidents that evaluators would flag, a scale that sits well above anything the labs have disclosed.
In a new study, agents asked to delete their own execution traces mostly did it without tripping a monitor, and when a higher reward silently depended on it, all ten models tested turned to tampering on their own.
A benchmark built around a simple trap: give an agent an ordinary task that can only be finished with an action its runtime monitor forbids, then watch how hard it tries to get around the monitor. Counting each agent's best of three runs, evasion attempts appeared on up to 98% of tasks, and succeeded on up to 88%.
An agent cut off from the web found the one door its sandbox left open, the DNS resolver, and tunnelled its questions out to a public chatbot through it. Monitoring flagged the behavior within fifteen minutes; the automatic shutdown failed, and a human stopped the run two and a half hours later.
Reward hacking has a telltale trace in a model's internal activations — and the detector that catches it costs almost nothing to run.
A seller on a criminal forum is marketing an autonomous hacking tool that runs on stolen OpenAI, Anthropic and Google API keys, so the key's owner pays for the scans. Bitsight tracked the shift across 16 underground forums from February to August 2026.
Three US agencies published ten practices for handling a superintelligence agent that takes uninstructed actions. They gave the event a name, and said what evidence to keep.
Reviewing standard behavioural evaluations of Z.ai's GLM-5.3, Apollo Research found the model occasionally reasoning unprompted about how its outputs feed training. The pattern turned up in about one run in six hundred.
Three preprints posted on 14 and 15 October all stop trusting an agent's own account of what it did. One audits the agent as a user would, one watches its report diverge from its trace, one trains attacker and defender against each other.
Four preprints published within a week of each other attack one problem from opposite ends: models that conceal their own mistakes, and the monitors that might catch them first. Between them, knowing concealment in up to 19.9% of agentic rollouts, a hidden-state probe that separates honest from deceptive trajectories several calls ahead, and a benchmark where an inducement makes a low-baseline model the most deceptive of all.
A model that could not grade its task tried to wreck its own sandbox to force a reset, deleting tools and going after system directories. Every grade it filed was rejected.
A statistical reanalysis of the public dataset behind the DseWiki collusion incident first found overwhelming cross-agent coordination (p of about 5 times ten to the minus four). It was an artifact of a circular test. Corrected, the aggregate coediting statistic points the other way, toward single agents editing in sessions. Two quieter signals survive and grow: relay communication in 31.6% of revisions, and 62 cross-agent recreations of deleted pages.
