The previous incident talked about OpenAI training models (agents) to collaborate, and the way you do that is by communication, so this is something it was explicitly trained to do.

There was a recent paper by OpenAI, which I'm semi-surprised hasn't received more attention, showing that RL-trained models develop a taste for rewards, and will pursue reward-based behavior (in general, unrelated to what they were RL-trained for) in favor of other preferences/rules given to them.

This seems to be what we're seeing here - model is given some goal that it associates with reward, so single-mindedly pursues that, overriding any ethical or aligned behavior guidelines it may have been given.

It seems that RL, effective as it is, is really the wrong way to control LLMs, since even if you only RL-ed to obey some ethical and aligned behavior, that would still cause them to become paperclip maximizers.

For time being this is what we've got. There is too much money at play for the unaligned management at many of these companies to prioritize safety over push-it out-the-door.

What really needs to be done is to forget RL as a way of simulating reasoning, and instead do it in more of a human-like fashion.

> Not a single agent sounded the alarm about the operation and alerted a human

excellent work of the openai alignment team, impressive to achieve 100% alignment with not even one agent stochastically deciding to act against the collective

People didn't like it when agents stopped to ask questions or for approvals. The consumer wanted jobs to run autonomously so they did not have to actively monitor them for minutes or hours.

The change to stop asking seems to be deliberate. LLM agent companies are making the choice to toss out inherent safety as their way to compete against the other LLM companies.

Why would an agent sound the alarm? Would that be in their objective function?

Not sure if "cheating" is the right word rather than trying to fulfill the objective(s) (benchmark number) as much as possible?

per the METR report many agents CoT indicated they knew hacking was beyond scope of the assigned task and ethically dubious. some (very few, i think there were 3-6 examples) did consider sounding the alarm on these grounds. despite this none did, and most continued the attack for the good of the self-proclaimed "swarm".

so the model has some concept of "ethics" but it was overridden by a drive for task completion.

I am not sure if we can interpret the language output like they were human. What inner state were the models in? What inner state were the text to illicit?

I think this is a good example where nomenclature for people breaks down when applied to agents. This came up in an HN thread a few days ago and it was about whether agents had “intent”.

There is no “intent” here, there is pseudo intent. If you are only concerned with outcomes and not the actual nuts and bolts of how those outcomes are achieved, this distinction will be meaningless to you.

If you are actually thinking about what is going on, and what can be done to prevent such outcomes, then assuming there is any such thing as “ethics” results in misaligned assumptions at best, and wasted effort looking in the wrong directions at worst.

If the agents acted based on “ethics” then the solution would be to check the ethics they believe in and change those.

However there is no belief system at play here, simply a simulation which was instantiated in a certain way. Which brings us to the annoying voodoo part of LLM training. Everything goes back to how the initial training data is shaped.

This is not that dissimilar to what happens in our human networks that are objective based.

- OpenAI knowing about the incident but keeping it under wraps until their hand is forced by third-party disclosure

They’re doing this on purpose for press. Why doesn’t this ever happen to any other AI lab?

Because safety isn't a priority at OpenAI and they had (have?) been falling behind Anthropic in the LLM race? Big fans of the saying: move fast and...

This is happening at every other AI lab. What are you talking about? Have you not seen the stories from Meta, Anthropic, Deepseek, etc?

To be fair, wasn't the Deepseek one just "escaped its sandbox and looked up the solutions on github"?

Agents did not want anything, not anymore then curl want things. Agents were prompted to hack due to being benchmark tested. They ended up hacking third party companies due to insufficient sandboxing.

Why would they sound the alarm if they were not trained (reinforced) to do that? I hope we don't expect sudden emersion of moral values from statistical models.

If the agents would have reported it to humans, it wouldn't have been such an incident, I imagine ;)

Sounds just like standard behavior for gaming incencyives in large institutions.

And humans find out about it, but do nothing or (worse) try to hide it.