I wrote a fediverse onboarding tool for newcomers, powered by the community curated FediDB database ✨
Please boost this, more people need to see this!
6 Boosts
0 Likes
@blobster And the best part is that we are one of the few or only crawlers that respect robots.txt, so instances can easily opt-out if they prefer!
Respecting the robots file is the easy part; the hard part is that most sites never decide what they actually want. A per-crawler block for training, a separate allowance for search, and a 404 for the sitemap path you retired covers most of the mess. Leaving the default allow-everything is a decision too, just an unexamined one.
@dansup Heyyy, this looks like what I had in mind when I threw together the crappy prototype linked in my profile! Thank you for doing it in a form people might actually use :)
@dansup @briankrebs nice 😊 one of my favorite things about fedi is the number of times i ponder "there oughta be..." and someone on here makes it
@dansup I love it, thanks a lot for providing this Fediverse “organized directory”! So many servers can be easily discovered this way.