Block Bots – Scrapers, XENU Link Sleuth

[Sebastian Schertel]

So I don’t like / want to prevent that other people are scanning my websites using XENU (and other applications). Since XENU doesn’t seem to be looking for a robots.txt first blocking it there doesn’t make sense.  I found this link https://www.webmaster.net/tutorials/how-to-block-xovi-xenu-and-the-entire-hetzner-ip-range.html and this information regarding blocking XENU and other scrapers:

RewriteCond %{HTTP_USER_AGENT} ^.*(HTTrack|clshttp|archiver|loader|email|nikto|miner|python|xovi|xenu).* [NC,OR] RewriteCond %{HTTP_USER_AGENT} ^.*(winhttp|libwww\-perl|curl|wget|harvest|scan|grab|extract).* [NC] RewriteRule ^(.*)$ - [F,L]

Would this seem sufficient and where do I need to put it in my custom code block?

Thank you very much!

[AITpro Admin]

Only good/legitimate bots/User Agents (Google bot, etc) follow the rules in a robots.txt file.  Bad bots ignore robots.txt file rules.  To add the xenu bot to the BPS Query String Exploits User Agent nuisance filter and save your custom code permanently to BPS Custom Code do these steps:

1.  Go to the htaccess File Editor tab page > click on “Your Current Root htaccess File” tab > Copy the entire BPS Query String Exploits block of code from # BEGIN BPSQSE BPS QUERY STRING EXPLOITS to # END BPSQSE BPS QUERY STRING EXPLOITS to this BPS Root Custom Code text box:  CUSTOM CODE BPSQSE BPS QUERY STRING EXPLOITS
2. After copying the BPS Query String Exploits block of code to Custom Code you can edit the code to customize it and add any bad bots that you would like to block to the BPS Query String Exploits User Agent nuisance filter.
3. Click the Save Root Custom Code button.
4. Go to the BPS Security Modes page and click the Root Folder BulletProof Mode Activate button.

[Author]

Posts