Discussion
This is fucking ridiculous. My https://openbenches.org/ website is serving over two TERABYTES of bandwidth per month!
That's up from 1.5TB the previous month.
While I'd love to believe that's all people fascinated by benches, I think it's more likely to be endless crawlers.
I'm lucky to have a very generous host with no hard bandwidth limits, but I'd want to find a way to block bots without having human-facing interstitials. Looks like robots.txt isn't cutting it.
@ryanc @outfrost @Edent I can't solve "client doesn't support any safe transport". 🤷 Until recently we did support plain http, but other than that I'm pretty sure it's TLS 1.2. I suppose older TLS or plain http could be offered on a separate use-at-your-own-risk port. If a user genuinely unable to access because of that contacted me, I would probably try to work something out for them.
I built a quick and dirty script to trap the bots which endlessly crawl my site.
I'm calling it "Blunt Banhammer Blocks Basic Bots Ballistically" - or B⁶ for short.
It has already dropped traffic from 200GB per day down to about 60GB per day.
Still nuts, of course, but slightly less nuts now.
Since yesterday, I've blocked nearly 100 IP addresses which had requested a hidden directory blocked by robots.txt - the graph already shows a dramatic fall in data transfer.
B⁶ is, as the name suggests, a blunt tool. But it's effective. I'll release the source code once I've tidied it up.
Some bots, like intricle-image-extractor are obvious scrapers, but most spoof common browsers. I half wish OSes would cryptographically sign their requests so you knew it came from a genuine device.
Mmm-hmm. All totally legitimate traffic. Nothing suspicious here, right?
On the 3rd of November, I implemented my "Blunt Banhammer Blocks Basic Bots Ballistically". Since then it has caught ~150 scrapers and cut ~75% of my traffic.
@Edent sure others have/will mention it, but try iocaine. It's catching the vast majority of bots on my site without causing issues for humans (that I'm aware of)
