The proliferation of AI crawlers is really making the blocklisting in robots.txt no longer practical since there is no robots.txt rule that can exclude all AI crawlers, each having a separate agent string.

On the other hand, only allowing large search engines is strengthening their dominant position.

I would prefer being able to qualify the usage rather than the agent.

Usage: “SearchEngine”
Disallow:

Usage: “TrainingData”
Disallow:/