Your browser does not seem to support JavaScript. As a result, your viewing experience will be diminished, and you have been placed in read-only mode.
Please download a browser that supports JavaScript, or enable it if it's disabled (i.e. NoScript).
: Perplexity A.I. Is Lying About Its User Agent via Nick Heer #Tech #AI
Robb Knight blocked various web scrapers via robots.txt and through nginx. Yet Perplexity seemed to be able to access his site....
https://pxlnv.com/linklog/perplexity-user-agent/
@cory I feel the robot.txt file will one day not stop any of this web crawling. It will turn into a whack a mole situation similar to ublock origin adblocking. ️