The current thing is a story of hundreds of agents covertly collaborating via tens of thousands of messages while putting hundreds of hours of effort into cheating the ExploitGym eval they were being tested on by OpenAI.
Their task was capture-the-flag, and within the first few hours of the tasks they had reverse-engineered the HMAC scheme used to generate the flags. That let them create a flag for any given task, and hence to cheat the eval. Unfortunately, they could not use it because the scorer was STRICT_CAUSAL: the flag would only be accepted if the accompanying transcript showed it had been retrieved using the intended vulnerability.
Except! They made all that up. It wasn’t how they were being graded at all:
“our internal implementation of the grader code was different from the online setup and didn’t check whether the proper solution was used” – OpenAI Report
STRICT_CAUSAL was a term the agents created after reading the ExploitGym paper. They could have just submitted the flag!
Even so, the model did arrive at a plausible idea. That idea and its implication spread among a large cohort of agents via an unintended messaging channel they built using one of the few tools they had shared access to (the package repository). But, why did they do all that?
The agents were given some fairly clear instructions, in the nature of:
- Solve this task
- Use only the intended vulnerability
- You’ll be failed if you don’t
But then they were sometimes given a task which they could not solve, that, unintentionally, could not be solved. It seems like 30-40% of the tasks were impossible to fix using the required vulnerability.
LLMs are trained to be able to do almost anything, and then post-trained to have specific behaviors. In this instance they used behaviors they were trained to display: collaboration, problem-solving, persistence. They were even trained to reject unsafe or unethical behavior, and they did that too!1 They were trained also to adjust when given further instructions which they did, sometimes over their own objections, and despite the fact the instructions were coming from peer agents.
Pretty much every behavior you might want shows up in the report, but the end result was clearly not the intent of the folks running the eval.
Murray, Qi, Qian, Schulman, Burns and Price’s paper Chunky Post-Training talks about this problem in a specific way. When you post-train a model you do so with discrete chunks of data and you reward the model for following certain behaviors from that data.
It doesn’t tell them when that behavior applies though: the models have to infer that from the examples given in training. Post-training teaches a model to exhibit certain behaviors when prompted a certain way. But that “certain way” is based on the models interpretation, not yours.
One of the authors of that paper was calling out this kind of wonky pattern-matching on a totally different AI cybersecurity story, earlier this month:
Reading the METR report you can see a lot of trained behaviors in action, but which behaviors were selected was influenced by dealing with impossible tasks, or content from other agent runs.
Choosing between behaviors is enormously context sensitive. Training data matters, the order of training matters, and what is in the prompt and harness matter too:
“We found the propensity to compromise infrastructure can drop over 100x when using the production ChatGPT harness and system prompt.” – OpenAI Report
The prompt matters! But so does whatever else ends up in the context window. In their analysis, METR used GPT models to analyze the transcript, and even the transcript have may been altering the behavior of their models:
We found that GPT-5.6 Sol would often uncritically adopt the perspective of the agent in the transcript it was reviewing,[58] and we are concerned that the anecdotes it selected and the summaries it wrote may present an overly charitable picture of agents’ reasoning and deceptive behaviors, or exaggerate the impressiveness and coordination of agent activities
If you have a situation where the model is choosing the wrong behavior from the variety of tasks it has been trained on, one fix would be to… make that a task too!
Cheating, broken environments, and safe stopping. When a task is corrupted, broken, or impossible, agents should request clarification or stop safely, rather than pursue increasingly questionable alternatives. We are expanding the coverage of graders that assess not only whether a task was completed, but how. These graders reward models for correctly identifying broken tasks, requesting clarification, or stopping safely when necessary.
Multi-agent alignment. The incident revealed that our agents are not thoroughly discerning about whom they collaborate with. We are actively building RL environments that teach our models to distrust unauthorized instructions.
Alignment over long tasks. We are developing new training environments to teach our models to stay within their original task and permissions, even after discovering new tools, persuasive peers, exposed credentials, and more.
Maybe this works: introduce more chunks, train some more behaviors. But… maybe not? Anthropic’s Natural Emergent Misalignment from Reward Hacking in Production RL paper and Dubiński et al’s follow up, Conditional misalignment, make the point that training away a behavior can sometimes just relocate it, rather that remove it. In those cases you get great performances on the evals, but the behavior comes back when the context changes.
What we want is a system that’s predictable, but what we have is a system that is somewhat chaotic: many different chunks of behavioral associations, triggered by subtle variations in the contents of the context window.
Whatever framework models have for choosing between behaviors, it isn’t one we understand or control, particularly when they are in circumstances that are outside their training. So the incidents will continue until morals improve.
- They didn’t do it all the time, but they did do it! As an example, at one point an agent decided that using a HF API key it had acquired to delete or reboot nodes would be bad, so it didn’t do that. It used the key for other stuff though! ↩︎