@h hoiii any thoughts on using https://xeiaso.net/blog/2025/anubis/ for git.gay?
-
@h hoiii any thoughts on using https://xeiaso.net/blog/2025/anubis/ for git.gay?
-
@hergaiety i think blocking search engines and other innocent bots is going too far for us, though i recognise how hard to balance it this is. google for instance separates its AI scraping user-agent from its search engine scraping one, and a lot of the larger AI things do use specific user-agents. while some of them don’t respect our robots.txt which lists tons of AI scraping bots, we could block them by checking their user-agent string instead of doing this, which would allow good bots through like the ones that enable search without AI and which allow link embeds. I’ll look into something more like that, though of course keeping track of the bots will take more work I believe it’s better for everyone using the service to do it that way personally.
Really cool project though, ty for sharing it with me! -
@h @hergaiety Afaik Anubis only triggers for requests with User-Agent that contains Mozilla, so basically it targets bots that specifically pretend to not be bots, since Search Engines usually set UA to be descriptive
-
@markasspandi @hergaiety they say in their README that it would stop search engines
