ROFLMAO.
-
ROFLMAO.
Claude decided to crawl one of the sites on my new server, where known bots are redirected to an iocaine maze. Claude has been in the maze for 13k requests so far, over the course of 30 minutes.
I will need to fine tune the rate limiting, because it didn't hit any rate limits - it scanned using 902 different client IPs. So simply rate limiting by IP doesn't fly. I'll rate limit by (possibly normalized) agent (they all used the same UA).
Over the course of this 30 minutes, it downloaded about ~300 times less data than if I would've let it scrape the real thing, and each request took about the tenth of the time to serve than the real thing would have. So I saved bandwidth, saved processing time, likely saved RAM too, and served garbage to Claude.
Job well done.
-
the vessel of morgannareplied to algernon ludd last edited by
@algernon 902 unique IPv4? Could you share some examples out of curiosity? I've got some access logs to peek at and compare against...
-
algernon luddreplied to the vessel of morganna last edited by
@astraleureka Yep, almost a thousand unique IPv4 addresses. I will write a post-mortem next weekend or so, which will include a whole lot of data. Will be posted on my blog, and will toot about it here too.
-
J [email protected] shared this topic
-
@algernon oh, I think I'll set this up on musk.sucks
-
O [email protected] shared this topic
