RE: https://mastodon.social/@fj/112280775190792281
Anthropic is proposing a related concept: Machine-Readable Training Preference Signal
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/
https://datatracker.ietf.org/doc/draft-kelley-aipref-training-preference-signal/
The proliferation of AI crawlers is really making the blocklisting in robots.txt no longer practical since there is no robots.txt rule that can exclude all AI crawlers, each having a separate agent string.
On the other hand, only allowing large search engines is strengthening their dominant position.
I would prefer being able to qualify the usage rather than the agent.
Usage: “SearchEngine”
Disallow:
Usage: “TrainingData”
Disallow:/
@fj
This all assumes that the AI companies are honest, and they are not.