Post by @scanlime@misc.name

lime with barcode

@scanlime@misc.name 2 hr. ago

today i learned there's a whole extra secret robots.txt just for "ai" scrapers, ".well-known/tdmrep.json"

the opt-out is:

[
{
"location": "/",
"tdm-reservation": 1
}
]

https://www.w3.org/community/reports/tdmrep/CG-FINAL-tdmrep-20240510/

TDM Reservation Protocol (TDMRep)

This specification defines a simple and practical Web protocol, capable of expressing the reservation of rights relative to text & data mining (TDM) applied to lawfully accessible Web content, and to ease the discovery of TDM licensing policies associated with such content.

0 Boosts

0 Likes

@arrjay@tacobelllabs.net 2 hr. ago

@scanlime ...and tdm-policy takes a uri itself.

this could be fun...

lime with barcode

@scanlime@misc.name 2 hr. ago

@arrjay my first impulse was to IP ban anyone who touches the file

lime with barcode

@scanlime@misc.name 2 hr. ago

@arrjay if they already scraped the data and then they're checking if they can "legally" use it though, sending a proper opt out may be more effective

@arrjay@tacobelllabs.net 1 hr. ago

@scanlime I was thinking proper opt-out, and basically classing the ip as "you can read the policy as many times as you like again and that's it"

1 more reply