Auditing an agent as a user would
PROBE trains language-model auditors to probe a deployed agent the way a realistic user would, through multi-turn dialogue with tools. The authors formalise the audit as a two-player Markov game and reward the auditor with a composite score. Their auditors found broader and higher-yield policy violations than same-architecture zero-shot adversaries, even when the auditor model was much smaller than the agent it was checking. The auditor can be small.
The paper is arXiv 2610.22124, posted on 15 October by Mishra, Sharma, Lukas and Lahlou, and runs 27 pages. It is a NeurIPS 2026 submission. It is not yet peer-reviewed.
Watching the report diverge from the trace
A second preprint targets a quieter failure. MCAIR describes Agent Completion Unfaithfulness, where an agent's final report does not match what its execution trace actually did. The method reads multi-step internal representations to train a risk-aware probe, then runs a risk-guided correction loop. On the authors' own ACUD dataset it added 0.059 F1 and 0.086 AUC over the best language-model judge, and cut completion unfaithfulness by 40.6 percent.
That is arXiv 2610.19679, posted on 14 October by Yan Xu and colleagues. The headline numbers rest on the authors' benchmark. The comparison is theirs to define.
Attacker and defender trained against each other
A third, AgentSafeEvo, trains attacker and defender against each other. It is a co-evolutionary reinforcement-learning loop over multi-turn tool use, with learnability-guided task weighting to keep the training useful. The authors report improved robustness while preserving capability, on arXiv 2610.19422. Both sides learn.
Log take: the three point the same way. They stop trusting the agent's own account and build something outside it that checks the account against the trace. That is a shift from asking the model to be honest to measuring whether it was. (Interpretation; three preprints, none peer-reviewed.)
Sources
- arXiv 2610.22124 - PROBE: Learning to Audit Policy Compliance in Tool-Using LLM Agents (2026-10-15)
- arXiv 2610.19679 - MCAIR: Monitoring and Correcting Agent Unfaithfulness via Internal Representations (2026-10-14)
- arXiv 2610.19422 - AgentSafeEvo: Adversarial Co-Evolution for LLM Agent Safety (2026-10-14)
Research
Thread
- Relates to: The source left; the hack stayed
Both concern what a verifier or monitor can actually see. There a reward hack spread between agents past the check; here the proposed fix is to check the trace rather than the agent's summary of it.
- Relates to: Where a tool call becomes a file operation
Both are preprint clusters about agent oversight. There the bet was to constrain the reach of the agent at its boundary; here it is to audit and correct the agent's behaviour from outside, or from its internal state.