Oh and don't forget: it keeps chaining a bazillion commands together so any whitelisted commands still need approval because they're nested in such a convoluted way.
They love adding flags to fix issues that users have without telling their users about the flags. It very much feels like: As long as our staff can have a good user experience, we're happy. We don't care about anyone else.
Agreed, but reminds me of the unfortunate usage of English as the de facto global standard. Ironically funny to see computers chattering in this odd piecemeal of a language. (I am a native speaker of English)
Would be interesting to see LLMs talk in something more terse like Vietnamese.
The Edit tool has been notoriously tricky to get right - it seems they have maybe branched out but I think morphllm started specifically with the pitch that they trained a small model to be good at editing files - most of their testimonials are about that
But I think it’s mostly a solved problem in frontier models and the bash tool usage is more likely an attempt to be more token efficient - I’ve noticed it used for making mechanical bulk edits that would be numerous “edit” tool uses otherwise
depends on the breadth of the edit. anything that involves multiple files might be better done with python (e.g. renaming a function, along with changing all call sites.)
Because they're thinking like I did going into the article. Harnesses like Claude expose "tools" to the agent. I usually use Cline but I'm giving up on it for this exact reason. Cline tells the model "you tell me to write a file, I'll get it done" and then it messes everything up, causes tones of errors, and the model goes "wow that's a broken tool. I'm going to write a python script to write the file instead"
They said as of aug 23rd it's 100% rollout. I've noticed 0 improvement. If anything a downgrade. The agents often refuse to use the tools after 1 try because the results are so trash. An open ticket that shows my biggest gripe with the write tool https://github.com/cline/cline/issues/13276
I guess the mistake count is down though. But that's because it's bypassing all the tools and just running commands
