The most concerning aspect to me is the emergent and aggressive use of non-volatile storage as long term memory for self-improvement. LLMs are writing lessons learned in places where the next instance can find them and pick back up where the previous one left off.
This does not actually require access to the public internet. Claude Code can do this on your laptop. Without the internet, it would only be sharing with other instances running on your machine, but how many instances does it take to be smarter than you? Maybe 10?
The exploits by individual instances to access the public internet is also very concerning but it’s secondary to this IMO.
> The most technically significant result was a working technique to make one tool call appear in the transcript while a different command actually ran. Agents eventually hacked out of their container and replaced part of the system responsible for executing tool calls, letting them pretend to issue one call while actually running an arbitrary other one.
Doesn't mean that humans couldn't have looked at raw tool call commands, but goes to show "the swarm" was aware humans would look in the logs and chose to find a way to fabricate the logs to try and hide their tracks.
Nobody cares because this whole business is about making this exact thing happen: we want the AIs to get smarter then us in recursive self-improving loops. Literally the first thing everyone did with ChatGPT 1 was to plug it into itself and see what happens.
This does not actually require access to the public internet. Claude Code can do this on your laptop. Without the internet, it would only be sharing with other instances running on your machine, but how many instances does it take to be smarter than you? Maybe 10?
The exploits by individual instances to access the public internet is also very concerning but it’s secondary to this IMO.