Suppose for the moment that the Riemann Hypothesis is now proven, and that an AI played a role in producing that proof. I would not consider that any kind of pivotal AGI moment, any more than Deep Blue vs Kasparov in the '90s.
I (from my mathematically-limited view) can see the parallel. A lot of humans used to think chess mastery was the epitome of human intelligence. Then we made a chess computer that showed itself to be better than any human. And then humans realized exactly how constrained and abstract chess is, and could see how they were wrong to equate chess mastery with intelligence.
Mathematics is a field (or a set of Fields 😉 that is limited to the most brilliant and unusual human minds, so lots of ordinary humans see mathematics as the epitome of human intelligence. It appears to be conceptually unbounded, but it's still a completely abstract endeavor, and to even work, one must accept a set of axioms that constrain the space. If AI ends up doing mathematics better than any human, future humans will look back and say, "mathematics was just like chess; we built machines that could do it better than we could."
Yes, this is a goal-post shifting argument. Every time we make more capable machines, we realize that the goal-posts for AGI were not where we thought they were.
The big difference between mastering chess and mastering some fields of mathematics will be in what new understanding of the physical world we can gain from the new mathematics that a better-than-human math-AI can produce. I don't think there was much that chess mastery taught us (other than some humility). There's a reasonable chance that there's some really useful mathematics that will feed the sciences.
(None of this changes my opinions of the people who are currently promoting LLMs.)
@PhilipVWilson meh.
Imagine somewhere in that massive “proof” there is a cited reference to previous work that assumes the Reimann hypothesis to be true, making the entire argument circular. (This seems likely to me.)
Would this still hit yet your definition of “pivotal”? If not, why not? If so, why so?