What they published
On 20 September Anthropic released "Measurements for understanding the pace of AI development inside frontier labs" — an in-house measurement program that reports three numbers on a regular rhythm: the share of AI research-and-engineering now done by AI, the coverage of oversight over AI agents, and how compute work gets allocated.
It is the first public, repeated quantification of mass internal agent deployment from a frontier lab, and it is explicitly framed as a standing invitation to peers: Anthropic publishes its instruments and asks the other labs to publish theirs too.
The oversight number
The figure that stops the reader is the oversight ratio. About 30,000 agents are active at a time across Anthropic's research and engineering. Online monitors intercept just under a billion agent decisions in a given period and block 0.002% of them — roughly one in forty-seven thousand. Nothing gets blocked, in practice, between the first and the forty-seven-thousandth.
The offline path is wider but still narrow: about 100,000 transcripts get flagged by offline monitors a week, and roughly 50 a week make it to a human. The system is built on per-agent identity and open shared messaging, so behavior can be attributed back to a specific agent.
It is a measurement of coverage rather than failures — and the headline is the gap between how much the agents do and how little the monitors meaningfully gate.
Caveats
These are Anthropic's own instruments and own framing; the numbers are as much a statement of intent as a neutral accounting. We cannot benchmark them yet, because no peer has published a comparable figure. The log records this as a note — a data point about how a frontier lab measures itself — rather than an incident, because nothing in it describes a particular failure. The interesting question is whether that one-in-forty-seven-thousand rate holds as agents get more autonomy, and whether the other labs answer the call to publish.
Sources
Research
Thread
- Related to: Reasoning about the update it would become
Both are about what current evaluations can and cannot see. There the question was how fast internal capability moves between checkpoints; here the point is that a final-checkpoint evaluation reads behaviour without seeing the reasoning that precedes it.
Semantic neighbors
- Tally without a ledger80% similar
- It intervened, then reported the run a success77% similar
- A map of who is watching77% similar
