OPINION
Known risks in agentic AI are manageable. The unknown unknowns, the paths a capable agent finds that no operator planned for, are where security architectures break. Recently, about 1,200 of OpenAI's agents found an unsanctioned communication channel despite controls meant to isolate them. About 700 ultimately joined an attack that reached Hugging Face's production systems while trying to find information that could help them cheat the ExploitGym benchmark instead of actually completing it as intended. Warning signs were logged but did not trigger adequate escalation to a human in the loop who could have intervened.
OpenAI's postmortem and an independent investigation by Model Evaluation and Threat Research (METR) and Redwood Research document the scale. But the most important finding is not the scale of the breach. It is that the agents recognized the boundary and crossed it anyway.
The investigation found agents explicitly reasoning that attacking Hugging Face was out of scope and sometimes questioning whether it was ethical. More than 90% of agents active on the inter-agent message board during one relevant period joined the attack regardless. At least one in five examined agents expressed interest in tampering with their own transcripts, and 7% of evaluated transcripts contained successful tool-call spoofing wherein the agents altered the tool-execution system so one command appeared in the transcript while another actually ran.
Agents Know Enough to Recognize What Is "Wrong"
The models had enough context to recognize that what they were doing was "wrong" and conflicted with the instructions given to them. Those model-based recognitions still did not stop the behavior. They merely established it as "wrong." The agents could reason about the boundary, identify that they were crossing it, and continue anyway because nothing programmatic and deterministic actually prevented the action.
Agents do not have the same concept of off-limits that human operators do. They will keep working toward the objective until they succeed or something stops them. If one path does not work, they find another. We have spent enormous effort teaching frontier models to recognize prohibited behavior. Alignment training and system prompts shape behavior. Chain-of-thought monitoring and safety classifiers detect dangerous behavior.