Pingdom – whitelist pingdom user agent/bot, allow pingdom user agent/bot

[silas88]

I have a bad bot user agent check (originally from Perishable Press) which is blocking pingdom from checking that my site is up.

RewriteCond %{HTTP_USER_AGENT} ^Ping [OR]

If I understand correctly
^Ping is effectively ^Ping* and so the good user agent Pingdom.com_bot_version_1.4_(http://www.pingdom.com) is also blocked.

How can I modify this rule so that only the pingdom user agent is allowed through? I was looking at some kind of expression based on negation of the user agent string e.g. ^Ping[^dom.com_bot*] but I am really not familiar enough with these rules to know if this is the best way to do this.

Also, once the RewriteEngine on directive is given can I delete all subsequent RewriteEngine on directives. I seem to have quite alot of them what with BPS code and my custom code.

For the sake of completeness the Pingdom recieves a 403 and this

Forbidden
You don’t have permission to access /wp-content/cache/supercache/mydomain.com//index.html on this server. Additionally, a 500 Internal Server Error error was encountered while trying to use an ErrorDocument to handle the request.

Strangely there is nothing about this in my BPS log. The Rewrite rule is

RewriteRule ^.* - [F,L]

[AITpro Admin]

I believe you would just need this RegEx – Ping[^dom] which means match Ping, but do not match d or o or m.  Even simpler is this:  Ping[^d] – just match Ping and not Pingd…

If you would like to allow HEAD Requests by bots on your website then…
1. Add this code below to this BPS Root Custom Code text box: CUSTOM CODE REQUEST METHODS FILTERED.
2. Click the Save Root Custom Code button.
3. Go to the BPS Security Modes page and click the Root Folder BulletProof Mode Activate button.

BPS Pro 11.6+ & BPS free .53.2+
You may see this code or the 11.5+/.53.1+ code in your root htaccess file.  The code does the same exact thing and is whitelisted in the same exact way.

# REQUEST METHODS FILTERED
# If you want to allow HEAD Requests use BPS Custom Code and copy
# this entire REQUEST METHODS FILTERED section of code to this BPS Custom Code
# text box: CUSTOM CODE REQUEST METHODS FILTERED.
# See the CUSTOM CODE REQUEST METHODS FILTERED help text for additional steps.
RewriteCond %{REQUEST_METHOD} ^(TRACE|DELETE|TRACK|DEBUG) [NC]
RewriteRule ^(.*)$ - [F]
#RewriteCond %{REQUEST_METHOD} ^(HEAD) [NC]
#RewriteRule ^(.*)$ /wp-content/plugins/bulletproof-security/405.php [L]

BPS Pro 11.5+ & BPS free .53.1+

# REQUEST METHODS FILTERED
# If you want to allow HEAD Requests use BPS Custom Code and copy
# this entire REQUEST METHODS FILTERED section of code to this BPS Custom Code
# text box: CUSTOM CODE REQUEST METHODS FILTERED.
# See the CUSTOM CODE REQUEST METHODS FILTERED help text for additional steps.
RewriteCond %{REQUEST_METHOD} ^(TRACE|DELETE|TRACK|DEBUG) [NC]
RewriteRule ^(.*)$ - [F]
#RewriteCond %{REQUEST_METHOD} ^(HEAD) [NC]
#RewriteRule ^(.*)$ - [R=405,L]

BPS Pro 11.4|BPS free .53 and lower versions

# REQUEST METHODS FILTERED
# If you want to allow HEAD Requests use BPS Custom Code and 
# remove/delete HEAD| from the Request Method filter.
# Example: RewriteCond %{REQUEST_METHOD} ^(TRACE|DELETE|TRACK|DEBUG) [NC]
# The TRACE, DELETE, TRACK and DEBUG Request methods should never be removed.
RewriteCond %{REQUEST_METHOD} ^(TRACE|DELETE|TRACK|DEBUG) [NC]
RewriteRule ^(.*)$ - [F]

[silas88]

Hmm, just found that * repeats the previous chachter 0 or more times, so maybe it should be something like this

RewriteCond %{HTTP_USER_AGENT} ^Ping[^dom\.com_bot.*] [OR]
RewriteRule ^.* - [F,L]

or instead should I put something like this in the TimThumb section

RewriteCond %{HTTP_USER_AGENT} ^Pingdom\.com_bot.* [NC]

This is so confusing!!! 🙂

P.S. Strange, I couldn’t find [OR] listed as a ReWrite rule flag in the apache user documentation

[AITpro Admin]

Yes, you could create an “is not” condition, but the Regex I posted previously should in theory work correctly.

RewriteCond %{HTTP_USER_AGENT} !^Pingdom

NOTE:  In order for Ping not to match Pingdom you would have to add a boundary of some sort, but with User Agent rules you have to allow the match to look at the rest of the User Agent String.  Example:  ^foobar$ does not work because the $ sign boundary will negate the User Agent String match.  And \bPing\b would not work either.

[silas88]

Hi AITpro,
Sorry I didn’t see your post there at first. Thanks for the suggestion. I don’t allow HEAD requests at the moment. Do you think if I white list it in the timthum section that would be a safe route to go, or is allowing HEAD requests also safe?

[AITpro Admin]

It is entirely up to you.  Allowing HEAD Requests does NOT have to do with security in any way.  This is just a nuisance filter.  See this similar post on whitelisting the UptimeRobot User Agent/bot:  http://forum.ait-pro.com/forums/topic/split-uptimerobot-whitelist-uptimerobot-bot/

What is important to note is that ONLY whitelisting by IP address works for whitelisting that bot.  Not sure why that is and whitelisting by User Agent/bot name does not work.  Or maybe adding uptimerobot.* would work because this would match the rest of the User Agent String, which is necessary when matching User Agents.

[silas88]

As it looks like I can whitelist this good bot based on the user agent I think I’ll continue to dissallow HEAD requests for the moment. I have this in the TimThumb section, need to test further after removing the comment from the original offending rule.

RewriteCond %{REQUEST_URI} (timthumb\.php|phpthumb\.php|thumb\.php|thumbs\.php) [NC]
RewriteCond %{HTTP_REFERER} ^.*mydomain.com.* [OR]
RewriteCond %{HTTP_USER_AGENT} ^Pingdom\.com_bot.* [NC]
RewriteRule . - [S=1]

Thanks for your guidance on this.

[silas88]

Well thinking about this some more I don’t think that position will work as later in the htaccess file it will get to the ^Ping bad bot rule and block it.

[AITpro Admin]

Does this simple rule work or not? Just match Ping and not Pingd…

^Ping[^d]

[silas88]

Yes, I was over complicating things completely.
KISS is best! 🙂

[AITpro Admin]

Yep, I tend to do that myself.  I was in the process of creating new Setup Wizard code and was saying to myself this code is bloated and ridiculous and there is a much better way to do this, but could not see that much better solution yet.  Got it now.  Sometimes you have to take the long road for a while until you see the shortcut.  😉

[AITpro Admin]

[Topic manually moved to this relevant Topic]
I followed the instructions that I found in this form – http://forum.ait-pro.com/forums/topic/pingdom-allow-through-bad-bots-custom-code/ to allow head requests but it’s still generating errors and blocking certain scripts when I run a speed test using pingdom tools. What else can I do to fix this?
__________________

Odd. It appears that things are working now. Perhaps it was my caching plugin that delayed the changes.

[Author]

Posts