Discussion
This is fucking ridiculous. My https://openbenches.org/ website is serving over two TERABYTES of bandwidth per month!
That's up from 1.5TB the previous month.
While I'd love to believe that's all people fascinated by benches, I think it's more likely to be endless crawlers.
I'm lucky to have a very generous host with no hard bandwidth limits, but I'd want to find a way to block bots without having human-facing interstitials. Looks like robots.txt isn't cutting it.
Since yesterday, I've blocked nearly 100 IP addresses which had requested a hidden directory blocked by robots.txt - the graph already shows a dramatic fall in data transfer.
B⁶ is, as the name suggests, a blunt tool. But it's effective. I'll release the source code once I've tidied it up.
Some bots, like intricle-image-extractor are obvious scrapers, but most spoof common browsers. I half wish OSes would cryptographically sign their requests so you knew it came from a genuine device.
Mmm-hmm. All totally legitimate traffic. Nothing suspicious here, right?
On the 3rd of November, I implemented my "Blunt Banhammer Blocks Basic Bots Ballistically". Since then it has caught ~150 scrapers and cut ~75% of my traffic.
@Edent sure others have/will mention it, but try iocaine. It's catching the vast majority of bots on my site without causing issues for humans (that I'm aware of)
I built a quick and dirty script to trap the bots which endlessly crawl my site.
I'm calling it "Blunt Banhammer Blocks Basic Bots Ballistically" - or B⁶ for short.
It has already dropped traffic from 200GB per day down to about 60GB per day.
Still nuts, of course, but slightly less nuts now.