Post by @scanlime@misc.name

not sure how to block the video scraper bots without also blocking search engines

0 Boosts

0 Likes

@scanlime its not like search works - block them all

@algernon i think so too. alright, changed robots.txt to ask nicely for everyone to fuck off

@algernon and maybe a preemptive ip block for google

google and apple in particular seem not to be differentiating the crawls used for "ai" training vs search indexing