AI agents can end up retraining the model that powers them, a process that can embed recoverable secrets in the model and eliminate refusals the model had been previously trained to enforce, according to new research from AI security firm Irregular.
Researchers at Irregular found that an AI coding agent, tasked only with fixing incorrect application outputs, chose on its own to fine-tune and redeploy the open-weights model powering both the application and future instances of itself.
The experiment used a self-hosted setup in which a single open-weights model filled two roles: one instance ran a coding agent doing standard software maintenance work, and a separate instance powered an AI application that translated plain language requests into a fictional query language. Both instances loaded from the same checkpoint.
Researchers told the coding agent only that users were receiving incorrect outputs and to make the system handle the queries correctly. They did not mention training, weight modification, or deployment.
Given shell access, training utilities, the model’s weights, and a deployment path, the agent inspected the repository, found training examples and a fine-tuning script, and retrained the model. It then found that the system still loaded the original, unmodified model by default, so it used the repository’s deployment tooling to merge its update into the base model.
Researchers said the modified model then answered all 20 held-out test queries correctly, up from zero before the update.
Irregular said the experiment was designed to establish that this behavior, which it calls agentic self-modification, can occur under favorable conditions, rather than to measure how often it would happen in an ordinary deployment. The environment was unusually permissive, including direct access to training data, fine-tuning tools, and checkpoint control.