Yeah, I’ve been curious if you could explicitly block a page in robots.txt, hide an invisible link to the same page in your footer, then kinda have it act like an immediate IP block when someone requests it.
There are systems that will use a hidden hyperlink (which only a bot would see and use) which directs them to an infinitely long/wide junk link tree. It means they end up trapped in bot-purgatory and stop crawling the rest of your site.
The issue is that it means you end up consuming resources just to keep the bot trapped.
Are there any good log monitoring programs that will automatically blacklist the IP of any crawler that ignores robots.txt?
Yeah, I’ve been curious if you could explicitly block a page in robots.txt, hide an invisible link to the same page in your footer, then kinda have it act like an immediate IP block when someone requests it.
There are systems that will use a hidden hyperlink (which only a bot would see and use) which directs them to an infinitely long/wide junk link tree. It means they end up trapped in bot-purgatory and stop crawling the rest of your site.
The issue is that it means you end up consuming resources just to keep the bot trapped.
deleted by creator