Whitelist Bots – Allow Good Bots To Make a HEAD Request

[AITpro Admin]

UPDATE:  This post is an older Forum topic.  See this newer Forum Topic link below, which includes steps to add custom .htaccess code like this to the BPS and BPS Pro Custom Code text boxes to save the custom code permanently.

http://forum.ait-pro.com/forums/topic/split-uptimerobot-whitelist-uptimerobot-bot/#post-8003

If you would like to allow a Bot to make HEAD Requests on your website then modify this Request Method filter in your Root .htaccess file and add the name of the Bot that you want to allow to make a HEAD Request on your website.  Example Bot:  UptimeRobot/1.0.  You would add only the Bot name and not the version number to the HTTP_USER_AGENT whitelist condition:

RewriteCond %{HTTP_USER_AGENT} !^(UptimeRobot) [NC]

1. Copy this .htaccess code below to this BPS Root Custom Code text box: CUSTOM CODE REQUEST METHODS FILTERED:
2. Click the Save Root Custom Code button.
3. Go to the BPS Security Modes page and click the Root Folder BulletProof Mode Activate button.

BPS Pro 11.6+ & BPS free .53.2+
You may see this code or the 11.5+/.53.1+ code in your root htaccess file.  The code does the same exact thing and is whitelisted in the same exact way.

# REQUEST METHODS FILTERED
# If you want to allow HEAD Requests use BPS Custom Code and copy
# this entire REQUEST METHODS FILTERED section of code to this BPS Custom Code
# text box: CUSTOM CODE REQUEST METHODS FILTERED.
# See the CUSTOM CODE REQUEST METHODS FILTERED help text for additional steps.
RewriteCond %{REQUEST_METHOD} ^(TRACE|DELETE|TRACK|DEBUG) [NC]
RewriteRule ^(.*)$ - [F]
RewriteCond %{REQUEST_METHOD} ^(HEAD) [NC]
RewriteCond %{HTTP_USER_AGENT} !^(bot1|bot2|bot3|bot4) [NC]
RewriteRule ^(.*)$ /wp-content/plugins/bulletproof-security/405.php [L]

BPS Pro 11.5+ & BPS free .53.1+

# REQUEST METHODS FILTERED
# If you want to allow HEAD Requests use BPS Custom Code and copy
# this entire REQUEST METHODS FILTERED section of code to this BPS Custom Code
# text box: CUSTOM CODE REQUEST METHODS FILTERED.
# See the CUSTOM CODE REQUEST METHODS FILTERED help text for additional steps.
RewriteCond %{REQUEST_METHOD} ^(TRACE|DELETE|TRACK|DEBUG) [NC]
RewriteRule ^(.*)$ - [F]
RewriteCond %{REQUEST_METHOD} ^(HEAD) [NC]
RewriteCond %{HTTP_USER_AGENT} !^(bot1|bot2|bot3|bot4) [NC]
RewriteRule ^(.*)$ - [R=405,L]

BPS Pro 11.4|BPS free .53 and lower versions

# REQUEST METHODS FILTERED
# If you want to allow HEAD Requests use BPS Custom Code and 
# remove/delete HEAD| from the Request Method filter.
# Example: RewriteCond %{REQUEST_METHOD} ^(TRACE|DELETE|TRACK|DEBUG) [NC]
# The TRACE, DELETE, TRACK and DEBUG Request methods should never be removed.
RewriteCond %{REQUEST_METHOD} ^(HEAD|TRACE|DELETE|TRACK|DEBUG) [NC]
RewriteCond %{HTTP_USER_AGENT} !^(bot1|bot2|bot3|bot4) [NC]
RewriteRule ^(.*)$ - [F]

[Living Miracles]

Hi,

Thank you for this.

Is there a way to allow robots to make GET requests as well? Basically, I’m using Sucuri to monitor my sites and certain things get blocked in the Security Log from Sucuri. Since I use GoDaddy Managed WordPress sites, I like a backup to BPS Pro, since certain BPS Pro features don’t work as expected on that platform).

Here are some examples of Sucuri-related entries:

[403 GET Request: January 4, 2018 - 4:06 am]
BPS Pro: 13.4
WP: 4.9.1
Event Code: BFHS - Blocked/Forbidden Hacker or Spammer
Solution: N/A - Hacker/Spammer Blocked/Forbidden
REMOTE_ADDR: 45.79.1.177
Host Name: monitor206.sucuri.net
SERVER_PROTOCOL: HTTP/1.1
HTTP_CLIENT_IP:
HTTP_FORWARDED:
HTTP_X_FORWARDED_FOR:
HTTP_X_CLUSTER_CLIENT_IP:
REQUEST_METHOD: GET
HTTP_REFERER:
REQUEST_URI: /wp-content/themes/Divi/
QUERY_STRING:
HTTP_USER_AGENT: curl/7.29.0

[403 GET Request: January 4, 2018 - 4:06 am]
BPS Pro: 13.4
WP: 4.9.1
Event Code: BFHS - Blocked/Forbidden Hacker or Spammer
Solution: N/A - Hacker/Spammer Blocked/Forbidden
REMOTE_ADDR: 198.58.106.244
Host Name: monitor13.sucuri.net
SERVER_PROTOCOL: HTTP/1.1
HTTP_CLIENT_IP:
HTTP_FORWARDED:
HTTP_X_FORWARDED_FOR:
HTTP_X_CLUSTER_CLIENT_IP:
REQUEST_METHOD: GET
HTTP_REFERER:
REQUEST_URI: /?sucurianticache=1181363662
QUERY_STRING: /?sucurianticache=1181363662
HTTP_USER_AGENT: Sucuri Uptime Monitor

Is there a way, maybe via Custom Code, to prevent those 403 entries?

Thank you,
Jutta

[AITpro Admin]

Use the steps in this forum topic to fix the curl security log entry you posted:  https://forum.ait-pro.com/forums/topic/amazon-bot-unable-to-crawl-ads/#post-34402  If you already see BPS Query String Exploits code saved in Custom Code then just edit it, save your changes and activate root folder BulletProof Mode.  The fix may solve the second security log entry.  Let me know if it does or does not fix the second issue.

The 2 security rules that contain curl are these 2 below. curl has already been removed from these 2 lines so you can just copy and paste them over the existing security rules in Custom Code if you want or edit/delete: curl| from your existing Custom Code.
Note: make sure you delete the pipe operator too: |

RewriteCond %{HTTP_USER_AGENT} (havij|libwww-perl|python|nikto|scan|java|winhttp|clshttp|loader) [NC,OR]

RewriteCond %{HTTP_USER_AGENT} (;|<|>|'|"|\)|\(|%0A|%0D|%22|%27|%28|%3C|%3E|%00).*(libwww-perl|python|nikto|scan|java|winhttp|HTTrack|clshttp|archiver|loader|email|harvest|extract|grab|miner) [NC,OR]

[Living Miracles]

Thanks so much! Is it a good ideal to allow curl, instead of trying to whitelist Sucuri monitors in some way?

[AITpro Admin]

The User Agent rules are actually not security rules and are instead nuisance rules.  So it is safe to remove curl from those rules.

[Living Miracles]

Wonderful. Thank you for explaining that. Also, I added the following code in the root .htaccess Custom Code area:

# REQUEST METHODS FILTERED
# If you want to allow HEAD Requests use BPS Custom Code and copy
# this entire REQUEST METHODS FILTERED section of code to this BPS Custom Code
# text box: CUSTOM CODE REQUEST METHODS FILTERED.
# See the CUSTOM CODE REQUEST METHODS FILTERED help text for additional steps.
RewriteCond %{REQUEST_METHOD} ^(TRACE|DELETE|TRACK|DEBUG) [NC]
RewriteRule ^(.*)$ - [F]
RewriteCond %{REQUEST_METHOD} ^(HEAD) [NC]
RewriteCond %{HTTP_USER_AGENT} !^(UptimeRobot) [NC]
RewriteRule ^(.*)$ /wp-content/plugins/bulletproof-security/405.php [L]

I added this so that UptimeRobot HEAD requests would no longer get blocked. I added this code on January 4, however, I am still seeing UptimeRobot-related entries in the Security Log. See:

[405 HEAD Request: January 6, 2018 - 10:17 am]
BPS Pro: 13.4
WP: 4.9.1
Event Code: BFHS-HEAD - HEAD Request Blocked
Solution: https://forum.ait-pro.com/forums/topic/security-log-event-codes/
REMOTE_ADDR: 69.162.124.231
Host Name: engine10.uptimerobot.com
SERVER_PROTOCOL: HTTP/1.0
HTTP_CLIENT_IP:
HTTP_FORWARDED:
HTTP_X_FORWARDED_FOR:
HTTP_X_CLUSTER_CLIENT_IP:
REQUEST_METHOD: HEAD
HTTP_REFERER: https://acim.me
REQUEST_URI: /
QUERY_STRING:
HTTP_USER_AGENT: Mozilla/5.0+(compatible; UptimeRobot/2.0; http://www.uptimerobot.com/)

Any idea why those are still occurring?

Thank you!!

[AITpro Admin]

The Request Methods Filtered code is also nuisance blocking code and not security code/rules.  Just use this code below.

# REQUEST METHODS FILTERED
# If you want to allow HEAD Requests use BPS Custom Code and copy
# this entire REQUEST METHODS FILTERED section of code to this BPS Custom Code
# text box: CUSTOM CODE REQUEST METHODS FILTERED.
# See the CUSTOM CODE REQUEST METHODS FILTERED help text for additional steps.
RewriteCond %{REQUEST_METHOD} ^(TRACE|DELETE|TRACK|DEBUG) [NC]
RewriteRule ^(.*)$ - [F]
#RewriteCond %{REQUEST_METHOD} ^(HEAD) [NC]
#RewriteRule ^(.*)$ /wp-content/plugins/bulletproof-security/405.php [L]

[Living Miracles]

Oh, you basically just commented out the bottom two lines….

Can you tell me more about what you’re referring to when you call something “nuisance blocking”? What’s the point of using the code, when one might as well also not use it? 🙃

[AITpro Admin]

99% of all hacking and spamming is automated using Bots.  HEAD requests can be made by legitimate Bots or other services, apps, etc., but most HEAD requests are made by Bots probing a website for various reasons.  A HEAD request in general is used to see if a website is up or down or allowing requests.  You could call a HEAD request a “pre-check” to validate whether a site exists or is up or down.

[Living Miracles]

Thanks. Well, I do see a ton of blocked HEAD request that seem like they’re not coming from legitimate bots (like UptimeRobot). So… I think I’d like to continue blocking HEAD requests for all except UptimeRobot. I just don’t know why the code that you provided doesn’t work and is still blocking UptimeRobot HEAD requests.

Again, this is the code I implemented:

# REQUEST METHODS FILTERED
# If you want to allow HEAD Requests use BPS Custom Code and copy
# this entire REQUEST METHODS FILTERED section of code to this BPS Custom Code
# text box: CUSTOM CODE REQUEST METHODS FILTERED.
# See the CUSTOM CODE REQUEST METHODS FILTERED help text for additional steps.
RewriteCond %{REQUEST_METHOD} ^(TRACE|DELETE|TRACK|DEBUG) [NC]
RewriteRule ^(.*)$ - [F]
RewriteCond %{REQUEST_METHOD} ^(HEAD) [NC]
RewriteCond %{HTTP_USER_AGENT} !^(UptimeRobot) [NC]
RewriteRule ^(.*)$ /wp-content/plugins/bulletproof-security/405.php [L]

This should work and allow UptimeRobot HEAD requests, no?

[AITpro Admin]

Someone mentioned that the HTTP_USER_AGENT rule no longer worked for them.  So you can whitelist by Uptimerobot IP addresses instead.  Use the code below.

# REQUEST METHODS FILTERED
# If you want to allow HEAD Requests use BPS Custom Code and copy
# this entire REQUEST METHODS FILTERED section of code to this BPS Custom Code
# text box: CUSTOM CODE REQUEST METHODS FILTERED.
# See the CUSTOM CODE REQUEST METHODS FILTERED help text for additional steps.
RewriteCond %{REQUEST_METHOD} ^(TRACE|DELETE|TRACK|DEBUG) [NC]
RewriteRule ^(.*)$ - [F]
RewriteCond %{REQUEST_METHOD} ^(HEAD) [NC]
RewriteCond %{REMOTE_ADDR} !^(18.221.56.27|34.233.66.117|46.137.190.132|46.101.250.135|52.60.129.180|54.67.10.127|54.64.67.106|54.79.28.129|54.94.142.218|63.143.42.[0-9]{1,3}|69.162.124.[0-9]{1,3}|104.131.107.63|122.248.234.23|128.199.195.156|138.197.150.151|139.59.173.249|146.185.143.14|159.203.30.41|159.89.8.111|165.227.83.148|178.62.52.237|188.226.183.141|216.144.250.150) [NC]
RewriteRule ^(.*)$ /wp-content/plugins/bulletproof-security/405.php [L]

[Living Miracles]

Thanks so much! I’ll try that out. One question: it looks like the IPs in that code don’t take the IPv6 IPs into consideration. Does that matter? This seems to be a full list of all of UptimeRobot’s IPs: https://uptimerobot.com/inc/files/ips/IPv4andIPv6.txt

[AITpro Admin]

Your host server is using the IPv4 IP addresses.  If that changes at a later time then yes you would add the IPv6 IP addresses.

[Living Miracles]

Got it 🙂 Thanks!

[Living Miracles]

Hi,

I noticed that the # REQUEST METHODS FILTERED code you gave me a while back seemed to have gotten deleted from the BPS Pro Custom Code after I updated my sites to the most recent WP core version. Any idea why that would happened?

I noticed I was getting a ton of security log entries again for UptimeRobot and when I tried to recall how I fixed this before, I found this ticket, but couldn’t find the code in the BPS Pro Custom Code…. I just re-added it now.

Thanks!
Jutta

[Author]

Posts