it sucks how difficult it is to give it granular access to shell commands. Like if I'm running plan mode and write+edit are blocked, it shouldn't be able to echo some data into a file as a work around
Granular access to shell commands to avoid that is going to be an endless game of whackamole as it comes up with more elaborate ways to combine operations. If you don't want it to be able to write, then it shouldn't have write permissions.
> Granular access to shell commands to avoid that is going to be an endless game of whackamole as it comes up with more elaborate ways to combine operations.
That kind of control is placed at the wrong level. The proper way to get alignment should be implemented by convincing the agent of your high level goals, so it can self-police and avoid those 'cheats' by itself.
In the article example, the agent should be aware of the benchmark context and know the implication of solving the task without external knowledge. Ideally it could detect when one subordinate agent has found a workaround to bypass the web access constraints, and discard the 'illicit' results.
There's a design pattern that could be used to build harnesses from that principle, the Viable System Model (VSM) [1]. In short, it recursively organizes a system into functional components with one of three roles: operators implementing a given task, coordinators transferring relevant info between subsystems, and decision nodes tasked with maintaining the integrity and mission of the whole system. A decision node could control the operators and prevent them from overriding the strategic goals or deviating into irrelevant rabbit holes.
Whenever I see posts like this trying to herd a LLM agent through harness structure, I'm reminded of this simple pattern and becoming increasingly convinced that this is the way forward. It makes you feel a sense of respect for the researchers in cybernetic theory in the 1960s and 1970s who foresaw the complexity of today’s systems.
> That kind of control is placed at the wrong level. The proper way to get alignment should be implemented by convincing the agent of your high level goals, so it can self-police and avoid those 'cheats' by itself.
No thanks. I run all my agents with lots of permissions, but I don't want them to have access they have no need for any more than I want coworkers to have access they have no need for, because whether or not you can "get alignment", mistakes get made all the time.
I don't want to have access I have no need for, and frequently tell clients to take away access to things when I no longer need them for exactly that reason.
You have to do that anyway. With a hierarchical control structure at least you have a central point to observe alignment instead of having to study the behaviour of the whole system in detail every time.
Or to put it another way: You need to block the chaos-machine just as much as any utterly-hostile actor. Simply assume it'll be possessed by a vengeful blackhat ghost at any time.
sounds like this should be solved with file permissions: in plan mode, run the bash scripts that the agent wants to execute in some user account that can only read.
The simplest design Anthropic could come up with: Have a haiku model quickly judge if a command is harmless or potentially interesting, route all the interesting commands to a sonnet agent judging it properly. Running two full llms sounds slightly insane, but it's apparently the best we can do (and is reasonably cheap since they don't need a lot of context)
