"AI" is a vague term that encompasses both traditional GOFAI, LLMs, and future tech so your question is meaningless. Non-human intelligence is possible, if that's what you're actually asking.

What part of my statement do you take issue with: that LLMs are pattern predictors (that's literally what the algorithm that runs it does) or that mathematics is rules-based and checkable and therefore amenable to automated pattern prediction?

My issue is that I can't predict what you'll find surprising, if an LLM (with a harness) would be able to do it.

If you are able to check that the results of an LLM are satisfactory, it means that the results are checkable. Then, in retrospect, the process of LLM coming up with those results is rule-based, because an LLM is a large set of data manipulation rules.

In short, which concrete thing that an LLM does would surprise you?

Perform the same tasks as a human can, given only the amount of training data and energy that a typical human has access to.

I guess it will take quite a while before we reverse engineer the human brain to find all the optimizations and shortcuts that evolution has used to make the human brain reach intellectual maturity in just about 20 years.