> And then you'd ask it about an area you're knowledgeable in and realise it routinely makes stupid mistakes.

This honestly doesn’t happen to me much anymore. In what areas do you find LLMs routinely make stupid mistakes?

Origami design will be my personal test bed for the coming years.

It's objectively very difficult and technical, it's spatiovisual, it's artistic, learning resources for it are sparse and most just learn by the FAFO method, current AI sucks terribly at it, and it's not likely to ever be specifically targeted by benchmaxxers.

In the last day of coding it has:

- Created useless pydantic schemas with all fields Optional[Any]

- Created a REST endpoint that silently mutated on GET (unsubscribed users from a mailing list)

- Failed to log costs in my app so users could have bankrupted me, etc, etc.

Good job I actually review its code.

It's really bad at game design

Scientifically useful physics simulations. Every model absolutely sucks at them.

Or, as someone else points out in another thread here, academic writing. It's one of the things newer models seem to have actually gotten worse at. Even when you give them detailed instructions on how to write and what to avoid, the "load-bearing", "A but not B" and journal-like writing make it in anyway, with the supposed AGI having no ability to reflect on how blatantly unacademic (and often unreadable) its writing is.