everytime i check nginx logs its more scrapers then i can count and i could not find any good open source solutions
You must log in or # to comment.
I’ve seen people suggesting and using Anubis, haven’t used it myself though.
deleted by creator
How do you know it’s “AI” scrappers?
I’ve have my server up before AI was a thing.
It’s totally normal to get thousands of bot hits and to get scraped.
I use crowdsec to mitigate it. But you will always get bot hits.
Some of them are at least honest and have it as a user agent.
Is ignoring robots.txt considered “honest”?
That’s not what I was talking about
