For the problems given in the article, it will 100% work. It's very easy for Claude 3.5 or gpt-4o to look at documentation for a couple of API endpoints, compare it to submitted code, and point out invalid endpoints and properties. It can provide correct code also if the custom is asking for something possible.
It won't be flawless but if the issues are as basic as stated in this article, then it seems like a good option.
It will cost money to use the good models though. So I think the first step would be to make sure if they ask for an invalid endpoint it says that in so many words in the API response, and if they ask for an invalid property it states that also.
Then if that doesn't give them a clue, an LLM scans incoming support requests and automatically answers them. Or flags them for review.. To prevent abuse, it might be a person doing it but just pressing a button to send an auto-answer.
But maybe a relatively cheap bot in a Discord channel or something connected to llama 3.1 70b would be good enough to start, and people have to provide a ticket number from that bot to get an answer from a person.
I'm a bit torn. My first thought was "If the current state of the art LLMs made the mistakes it's unlikely an LLM would be able to correct them." But I'm not sure that's true if the support LLM (chat bot) is given very specific instructions so as to limit the possible answers. Still I think that's gonna break down pretty quick for other reasons.
Maybe the chat bot can recognize a botched request and even suggest a fix but what then? It certainly won't be able to convert the user's next request into a working application of even moderate complexity. And do you really want a chat bot to be the first interaction your customers have.
I think this is why we haven't seen these things take off outside of very large organizations who are looking to save money in exchange for making customers ask for a human when they need one.
> I'm a bit torn. My first thought was "If the current state of the art LLMs made the mistakes it's unlikely an LLM would be able to correct them."
But, I mean, that doesn't make sense even for humans, right? 99% of the errors I make, I can easily correct myself because they're trivial. But you still have to go through the process of fixing them, it doesn't happen on its own. Like, for instance, just now I typoed "stil" and had to backspace a few letters to fix it. But LLMs cannot backspace (!), so you have to put them into a context where they feel justified in going back and re-typing their previous code.
That's why it's a bit silly to make LLM code, try to run it, see an error and immediately run to the nearest human. At least let the AI do a few rounds of "Can you spot any problems or think of improvements?" first!
