Managing Crawlers - ChatGPT
-
Is anyone looking at the ChatGPT and similar crawlers hitting their servers in terms of load?
If so have you put in place mitigations and do they work?
-
I've opened a PR on the dependency we use so we can at least detect gptbot and deny access that way.
feat: add GPTBot by barisusakli · Pull Request #15 · binarykitchen/spider-detector
A tiny node module to detect spiders/crawlers quickly and comes with optional middleware for ExpressJS - feat: add GPTBot by barisusakli · Pull Request #15 · binarykitchen/spider-detector
GitHub (github.com)
-
I propose we classify this traffic as a
synthetic-entity
traffic, if anyone has a better handle on the correct classification term, speak now.
Copyright © 2024 NodeBB | Contributors