It's not that different, when a human proposes a better definition vis-a-vis a competing one for example, they would defend this by certain desiderata.
Often a mathematician or physicist will use their intuition to speed up the naive brute force of candidate well formed formula variations so that the desired properties emerge, postulating the existence of an intersection on multiple desiderata can in itself be viewed as a novel conjecture, to be proven or disproved.
A very basic (unimpressive) example for an example desideratum is regularity or compactness. the tau=2 * pi substitution does make a whole bunch of expressions more slightly more regular and compact. That is something objective and measurable on a system of theorems.
There is no mathematician's moat vis-a-vis machine learning at a fundamental level. There can be artificially sustained moat, if AI powers limit the distribution of say cryptographic advance capable models, in jurisdictions outside such AI powers, but even that would be expected to be fleeting and temporary...
You're conflating a discussion about current LLM capabilities with your fantasies about nonexistent future AI. LLMs act nothing like this, and the small example you're giving is only a small part of the things that LLMs can't do.
Not that I'm a professional mathematician, but I'm not seeing why people think definitions are somehow a blocker. LLMs have no issue making definitions (interfaces/traits/abstract classes) in programming, which is formally the same activity. I ask them to form a core "spine" of a program (basically an intelligible theory), and they do it really well.
Like when we had these recent counterexamples to various conjectures, it's then pretty obvious to say "okay why did that counterexample work when most examples people looked at didn't" or equivalently "characterize examples that work vs examples that don't". There's your definition. "Def: An 'evil' polynomial is one that... Thm: conjecture is true iff f is non-evil. Thm: f is evil iff f is dastardly and a menace. "
Or if you think it won't be able to come up with a sufficiently good name, just tell it to call the happy case normal, and it will be in good company with humans[0]. Sprinkle in some semi-, quasi-, pre-, and para- to cover the various different ways the thing might satisfy some but not all properties of being normal, and it'll fit right in. "A polynomial is of quasiprenormal Claude type if..."
> LLMs have no issue making definitions (interfaces/traits/abstract classes) in programming, which is formally the same activity. I ask them to form a core "spine" of a program (basically an intelligible theory), and they do it really well.
Is it a "standard" software? Something where the patterns exists in several other software? Try with something that is novel, or is in a limited set. You will find that it will copy heavily from what exists already, going so far as lifting whole functions from another project.
The goalpost moving is really getting absurd, to the point where now the machine needs to be a world-class once-a-century genius that invents entire new fields out of thin air (which are of course still relevant to humans) for it to be "intelligent". Meanwhile a well above average human struggles to even apply trivial definitions to particular problems (c.f. programmers that don't understand monoids).
Not PC but is ideologically high-handed and inappropriate to accuse the other side of failing to stick to your desired framing of a discussion.
Computer science is about what LLMs fundamentally are. If you implicitly focus on "actually existing LLMs", and require others do this, then that is not computer science. That is politics.
I think the parent meant it is more interesting to pose new problems than solve them. Posing a new conjecture along the path to solving something is a close cousin, but still seems more bounded than proposing something novel to prove—if only because proving that something novel is also actually interesting is subjective and thus difficult for a different reason.
