Whitelist Bots – Allow Good Bots To Make a HEAD Request

Home Forums BulletProof Security Pro Whitelist Bots – Allow Good Bots To Make a HEAD Request

This topic contains 16 replies, has 2 voices, and was last updated by  Living Miracles 2 weeks, 5 days ago.

Viewing 15 posts - 1 through 15 (of 17 total)
  • Author
    Posts
  • #1427

    AITpro Admin
    Keymaster

    UPDATE:  This post is an older Forum topic.  See this newer Forum Topic link below, which includes steps to add custom .htaccess code like this to the BPS and BPS Pro Custom Code text boxes to save the custom code permanently.

    http://forum.ait-pro.com/forums/topic/split-uptimerobot-whitelist-uptimerobot-bot/#post-8003

    If you would like to allow a Bot to make HEAD Requests on your website then modify this Request Method filter in your Root .htaccess file and add the name of the Bot that you want to allow to make a HEAD Request on your website.  Example Bot:  UptimeRobot/1.0.  You would add only the Bot name and not the version number to the HTTP_USER_AGENT whitelist condition:

    RewriteCond %{HTTP_USER_AGENT} !^(UptimeRobot) [NC]

    1. Copy this .htaccess code below to this BPS Root Custom Code text box: CUSTOM CODE REQUEST METHODS FILTERED:
    2. Click the Save Root Custom Code button.
    3. Go to the BPS Security Modes page and click the Root Folder BulletProof Mode Activate button.

    BPS Pro 11.6+ & BPS free .53.2+
    You may see this code or the 11.5+/.53.1+ code in your root htaccess file.  The code does the same exact thing and is whitelisted in the same exact way.

    # REQUEST METHODS FILTERED
    # If you want to allow HEAD Requests use BPS Custom Code and copy
    # this entire REQUEST METHODS FILTERED section of code to this BPS Custom Code
    # text box: CUSTOM CODE REQUEST METHODS FILTERED.
    # See the CUSTOM CODE REQUEST METHODS FILTERED help text for additional steps.
    RewriteCond %{REQUEST_METHOD} ^(TRACE|DELETE|TRACK|DEBUG) [NC]
    RewriteRule ^(.*)$ - [F]
    RewriteCond %{REQUEST_METHOD} ^(HEAD) [NC]
    RewriteCond %{HTTP_USER_AGENT} !^(bot1|bot2|bot3|bot4) [NC]
    RewriteRule ^(.*)$ /wp-content/plugins/bulletproof-security/405.php [L]

    BPS Pro 11.5+ & BPS free .53.1+

    # REQUEST METHODS FILTERED
    # If you want to allow HEAD Requests use BPS Custom Code and copy
    # this entire REQUEST METHODS FILTERED section of code to this BPS Custom Code
    # text box: CUSTOM CODE REQUEST METHODS FILTERED.
    # See the CUSTOM CODE REQUEST METHODS FILTERED help text for additional steps.
    RewriteCond %{REQUEST_METHOD} ^(TRACE|DELETE|TRACK|DEBUG) [NC]
    RewriteRule ^(.*)$ - [F]
    RewriteCond %{REQUEST_METHOD} ^(HEAD) [NC]
    RewriteCond %{HTTP_USER_AGENT} !^(bot1|bot2|bot3|bot4) [NC]
    RewriteRule ^(.*)$ - [R=405,L]

    BPS Pro 11.4|BPS free .53 and lower versions

    # REQUEST METHODS FILTERED
    # If you want to allow HEAD Requests use BPS Custom Code and 
    # remove/delete HEAD| from the Request Method filter.
    # Example: RewriteCond %{REQUEST_METHOD} ^(TRACE|DELETE|TRACK|DEBUG) [NC]
    # The TRACE, DELETE, TRACK and DEBUG Request methods should never be removed.
    RewriteCond %{REQUEST_METHOD} ^(HEAD|TRACE|DELETE|TRACK|DEBUG) [NC]
    RewriteCond %{HTTP_USER_AGENT} !^(bot1|bot2|bot3|bot4) [NC]
    RewriteRule ^(.*)$ - [F]
    #35215

    Living Miracles
    Participant

    Hi,

    Thank you for this.

    Is there a way to allow robots to make GET requests as well? Basically, I’m using Sucuri to monitor my sites and certain things get blocked in the Security Log from Sucuri. Since I use GoDaddy Managed WordPress sites, I like a backup to BPS Pro, since certain BPS Pro features don’t work as expected on that platform).

    Here are some examples of Sucuri-related entries:

    [403 GET Request: January 4, 2018 - 4:06 am]
    BPS Pro: 13.4
    WP: 4.9.1
    Event Code: BFHS - Blocked/Forbidden Hacker or Spammer
    Solution: N/A - Hacker/Spammer Blocked/Forbidden
    REMOTE_ADDR: 45.79.1.177
    Host Name: monitor206.sucuri.net
    SERVER_PROTOCOL: HTTP/1.1
    HTTP_CLIENT_IP:
    HTTP_FORWARDED:
    HTTP_X_FORWARDED_FOR:
    HTTP_X_CLUSTER_CLIENT_IP:
    REQUEST_METHOD: GET
    HTTP_REFERER:
    REQUEST_URI: /wp-content/themes/Divi/
    QUERY_STRING:
    HTTP_USER_AGENT: curl/7.29.0

    [403 GET Request: January 4, 2018 - 4:06 am]
    BPS Pro: 13.4
    WP: 4.9.1
    Event Code: BFHS - Blocked/Forbidden Hacker or Spammer
    Solution: N/A - Hacker/Spammer Blocked/Forbidden
    REMOTE_ADDR: 198.58.106.244
    Host Name: monitor13.sucuri.net
    SERVER_PROTOCOL: HTTP/1.1
    HTTP_CLIENT_IP:
    HTTP_FORWARDED:
    HTTP_X_FORWARDED_FOR:
    HTTP_X_CLUSTER_CLIENT_IP:
    REQUEST_METHOD: GET
    HTTP_REFERER:
    REQUEST_URI: /?sucurianticache=1181363662
    QUERY_STRING: /?sucurianticache=1181363662
    HTTP_USER_AGENT: Sucuri Uptime Monitor

    Is there a way, maybe via Custom Code, to prevent those 403 entries?

    Thank you,
    Jutta

    #35217

    AITpro Admin
    Keymaster

    Use the steps in this forum topic to fix the curl security log entry you posted:  https://forum.ait-pro.com/forums/topic/amazon-bot-unable-to-crawl-ads/#post-34402  If you already see BPS Query String Exploits code saved in Custom Code then just edit it, save your changes and activate root folder BulletProof Mode.  The fix may solve the second security log entry.  Let me know if it does or does not fix the second issue.

    The 2 security rules that contain curl are these 2 below. curl has already been removed from these 2 lines so you can just copy and paste them over the existing security rules in Custom Code if you want or edit/delete: curl| from your existing Custom Code.
    Note: make sure you delete the pipe operator too: |

    RewriteCond %{HTTP_USER_AGENT} (havij|libwww-perl|python|nikto|scan|java|winhttp|clshttp|loader) [NC,OR]
    
    RewriteCond %{HTTP_USER_AGENT} (;|<|>|'|"|\)|\(|%0A|%0D|%22|%27|%28|%3C|%3E|%00).*(libwww-perl|python|nikto|scan|java|winhttp|HTTrack|clshttp|archiver|loader|email|harvest|extract|grab|miner) [NC,OR]
    #35222

    Living Miracles
    Participant

    Thanks so much! Is it a good ideal to allow curl, instead of trying to whitelist Sucuri monitors in some way?

    #35223

    AITpro Admin
    Keymaster

    The User Agent rules are actually not security rules and are instead nuisance rules.  So it is safe to remove curl from those rules.

    #35226

    Living Miracles
    Participant

    Wonderful. Thank you for explaining that. Also, I added the following code in the root .htaccess Custom Code area:

    # REQUEST METHODS FILTERED
    # If you want to allow HEAD Requests use BPS Custom Code and copy
    # this entire REQUEST METHODS FILTERED section of code to this BPS Custom Code
    # text box: CUSTOM CODE REQUEST METHODS FILTERED.
    # See the CUSTOM CODE REQUEST METHODS FILTERED help text for additional steps.
    RewriteCond %{REQUEST_METHOD} ^(TRACE|DELETE|TRACK|DEBUG) [NC]
    RewriteRule ^(.*)$ - [F]
    RewriteCond %{REQUEST_METHOD} ^(HEAD) [NC]
    RewriteCond %{HTTP_USER_AGENT} !^(UptimeRobot) [NC]
    RewriteRule ^(.*)$ /wp-content/plugins/bulletproof-security/405.php [L]

    I added this so that UptimeRobot HEAD requests would no longer get blocked. I added this code on January 4, however, I am still seeing UptimeRobot-related entries in the Security Log. See:

    [405 HEAD Request: January 6, 2018 - 10:17 am]
    BPS Pro: 13.4
    WP: 4.9.1
    Event Code: BFHS-HEAD - HEAD Request Blocked
    Solution: https://forum.ait-pro.com/forums/topic/security-log-event-codes/
    REMOTE_ADDR: 69.162.124.231
    Host Name: engine10.uptimerobot.com
    SERVER_PROTOCOL: HTTP/1.0
    HTTP_CLIENT_IP:
    HTTP_FORWARDED:
    HTTP_X_FORWARDED_FOR:
    HTTP_X_CLUSTER_CLIENT_IP:
    REQUEST_METHOD: HEAD
    HTTP_REFERER: https://acim.me
    REQUEST_URI: /
    QUERY_STRING:
    HTTP_USER_AGENT: Mozilla/5.0+(compatible; UptimeRobot/2.0; http://www.uptimerobot.com/)

    Any idea why those are still occurring?

    Thank you!!

    #35227

    AITpro Admin
    Keymaster

    The Request Methods Filtered code is also nuisance blocking code and not security code/rules.  Just use this code below.

    # REQUEST METHODS FILTERED
    # If you want to allow HEAD Requests use BPS Custom Code and copy
    # this entire REQUEST METHODS FILTERED section of code to this BPS Custom Code
    # text box: CUSTOM CODE REQUEST METHODS FILTERED.
    # See the CUSTOM CODE REQUEST METHODS FILTERED help text for additional steps.
    RewriteCond %{REQUEST_METHOD} ^(TRACE|DELETE|TRACK|DEBUG) [NC]
    RewriteRule ^(.*)$ - [F]
    #RewriteCond %{REQUEST_METHOD} ^(HEAD) [NC]
    #RewriteRule ^(.*)$ /wp-content/plugins/bulletproof-security/405.php [L]
    #35228

    Living Miracles
    Participant

    Oh, you basically just commented out the bottom two lines….

    Can you tell me more about what you’re referring to when you call something “nuisance blocking”? What’s the point of using the code, when one might as well also not use it? 🙃

    #35229

    AITpro Admin
    Keymaster

    99% of all hacking and spamming is automated using Bots.  HEAD requests can be made by legitimate Bots or other services, apps, etc., but most HEAD requests are made by Bots probing a website for various reasons.  A HEAD request in general is used to see if a website is up or down or allowing requests.  You could call a HEAD request a “pre-check” to validate whether a site exists or is up or down.

    #35230

    Living Miracles
    Participant

    Thanks. Well, I do see a ton of blocked HEAD request that seem like they’re not coming from legitimate bots (like UptimeRobot). So… I think I’d like to continue blocking HEAD requests for all except UptimeRobot. I just don’t know why the code that you provided doesn’t work and is still blocking UptimeRobot HEAD requests.

    Again, this is the code I implemented:

    # REQUEST METHODS FILTERED
    # If you want to allow HEAD Requests use BPS Custom Code and copy
    # this entire REQUEST METHODS FILTERED section of code to this BPS Custom Code
    # text box: CUSTOM CODE REQUEST METHODS FILTERED.
    # See the CUSTOM CODE REQUEST METHODS FILTERED help text for additional steps.
    RewriteCond %{REQUEST_METHOD} ^(TRACE|DELETE|TRACK|DEBUG) [NC]
    RewriteRule ^(.*)$ - [F]
    RewriteCond %{REQUEST_METHOD} ^(HEAD) [NC]
    RewriteCond %{HTTP_USER_AGENT} !^(UptimeRobot) [NC]
    RewriteRule ^(.*)$ /wp-content/plugins/bulletproof-security/405.php [L]

    This should work and allow UptimeRobot HEAD requests, no?

    #35231

    AITpro Admin
    Keymaster

    Someone mentioned that the HTTP_USER_AGENT rule no longer worked for them.  So you can whitelist by Uptimerobot IP addresses instead.  Use the code below.

    # REQUEST METHODS FILTERED
    # If you want to allow HEAD Requests use BPS Custom Code and copy
    # this entire REQUEST METHODS FILTERED section of code to this BPS Custom Code
    # text box: CUSTOM CODE REQUEST METHODS FILTERED.
    # See the CUSTOM CODE REQUEST METHODS FILTERED help text for additional steps.
    RewriteCond %{REQUEST_METHOD} ^(TRACE|DELETE|TRACK|DEBUG) [NC]
    RewriteRule ^(.*)$ - [F]
    RewriteCond %{REQUEST_METHOD} ^(HEAD) [NC]
    RewriteCond %{REMOTE_ADDR} !^(18.221.56.27|34.233.66.117|46.137.190.132|46.101.250.135|52.60.129.180|54.67.10.127|54.64.67.106|54.79.28.129|54.94.142.218|63.143.42.[0-9]{1,3}|69.162.124.[0-9]{1,3}|104.131.107.63|122.248.234.23|128.199.195.156|138.197.150.151|139.59.173.249|146.185.143.14|159.203.30.41|159.89.8.111|165.227.83.148|178.62.52.237|188.226.183.141|216.144.250.150) [NC]
    RewriteRule ^(.*)$ /wp-content/plugins/bulletproof-security/405.php [L]
    
    
    		
    	
    #35233

    Living Miracles
    Participant

    Thanks so much! I’ll try that out. One question: it looks like the IPs in that code don’t take the IPv6 IPs into consideration. Does that matter? This seems to be a full list of all of UptimeRobot’s IPs: https://uptimerobot.com/inc/files/ips/IPv4andIPv6.txt

    #35235

    AITpro Admin
    Keymaster

    Your host server is using the IPv4 IP addresses.  If that changes at a later time then yes you would add the IPv6 IP addresses.

    #35236

    Living Miracles
    Participant

    Got it 🙂 Thanks!

    #35725

    Living Miracles
    Participant

    Hi,

    I noticed that the # REQUEST METHODS FILTERED code you gave me a while back seemed to have gotten deleted from the BPS Pro Custom Code after I updated my sites to the most recent WP core version. Any idea why that would happened?

    I noticed I was getting a ton of security log entries again for UptimeRobot and when I tried to recall how I fixed this before, I found this ticket, but couldn’t find the code in the BPS Pro Custom Code…. I just re-added it now.

    Thanks!
    Jutta

Viewing 15 posts - 1 through 15 (of 17 total)

You must be logged in to reply to this topic.