Discussion
okay it looks like my at-home server is constantly sending around 40Mbps to my linode over tailscale, and I'm having trouble figuring out where the traffic is coming from. iptraf and iftop tell me that the data transfer is happening but can't tell me which processes are involved.
Anyone have any ideas?
alright I think I figured it out, a bunch of fucking bots were crawling my entire peertube instance, including all of the video files, so yeah of course that's gonna chew up a lot of bandwidth. I added 'Disallow: /' to robots.txt on that and hopefully that'll eventually improve things
also the Disallow: / should be the fucking default, not something buried 10 pages down in shitty documentation
anyway thanks to @scanlime I now also have ACLs set up to make it easy to block badly-behaving AI crawlers from hammering my peertube instance as well. hopefully this gets my linode out of alarm status.
@fluffy oh no.. first figure out which process is responsible, use the port numbers from tcpdump/etc with netstat to find process, then check that process logs. i'm struggling with the bandwidth caps on linode and 40Mbps is not happy. i've been adding nginx deny lines or similar for hosts like that.
@scanlime It's the "figure out which process is responsible" that I'm having trouble with
@fluffy sorry i could have been more clear. i usually use something like "sudo netstat -lutnp" to quickly see what's listening. if you need to, capture a traffic sample with wireshark or tcpdump and use the port number to be sure what's what.
@fluffy if it's not listen but outgoing traffic, you can use netstat to look for that too but it might be harder if it's ephemeral
@scanlime tcpdump -i tailscale0 is a bit more informative
@scanlime but it's just saying it's https traffic and like, I figured, because that's what all my tunneled services are doing, but /var/log/nginx/access.log isn't telling me anything useful
@fluffy another tool that might help, wireshark/tshark should be able to extract SNI/hostname from the handshake if you can capture initial tls connection
@scanlime as far as I can tell the issue is just a constant flood of 'POST /inbox' on my peertube instance
@fluffy from one host or lots of hosts? that flood is somewhat normal but it shouldn't add up to 40 Mbits
@scanlime unclear, unfortunately it all appears to be coming from my fronting linode, although the user-agent says it's mostly toot.community
@fluffy oh yeah, if you are looking at loopback you will need a tool that parses http headers to see what traffic comes from what host
@scanlime yeah I just changed nginx to log the forwarded_for address instead and it's a bit more useful
what's weird is that it seems that 99% of my traffic is coming from toot.community which is supposedly shutting down? and is also not a peertube instance to begin with? maybe I should just block it :/
@scanlime okay I think applebot is indexing my peertube server and downloading all the videos or something? that'd certainly explain things :|
@fluffy dang, i got hit by that too. sorry.
root@tuber:~# cat /etc/nginx/snippets/video-acl.conf
include /etc/nginx/snippets/obverse-acl.conf;
# each of these was motivated by a scraper downloading every video
# in bulk as fast as my server can handle. fuck "ai".
# microsoft
deny 20.118.222.0/24;
# apple
deny 17.166.0.0/16;
# google
deny 66.249.64.0/19;
# facebook
deny 2a03:2880::/29;
@scanlime in tcpdump I just see a shitload of UDP packets which I assume are tailscale tunneling
@fluffy Linux? `ss` is the tool you need
@raggi linux yes, thanks, I'll try that out
@raggi hm, this doesn't seem to do much more than netstat by default. is there a usage guide for the actual "which socket is sending a lot of data" thing?
the issue with iftop is it only tells me what the remote address is but doesn't do any breakdown based on socket, for example. if I knew the actual TCP port then I could trivially lsof it.
I need to look into using `tailscale serve` instead of fronting everything on nginx on linode but I'm not sure how to set that up becasue as always DOCS FUCKING SUCK