Googlebot – Googlebot-Image 403 error

[AITpro Admin]

This is a common known issue: The Googlebot is able to retrieve images successfully, but something else that the script is doing (HEAD Request) in combination with a plugin or theme coding on your site that handles image files also triggers a 403 error. You can just choose to ignore these erorrs on the Security Log page by ignoring the Googlebot User Agent (means not logging errors-NOT ignore the Googlebot) or you can whitelist the Googlebot. The choice is up to you.

Whitelist approach…

Another approach would be to whitelist the Googlebot User Agent. You would add RewriteCond %{HTTP_USER_AGENT} ^(.*Googlebot.*) to this security filter in your root .htaccess file.

# TIMTHUMB FORBID RFI and MISC FILE SKIP/BYPASS RULE
# Use BPS Custom Code to modify/edit/change this code and to save it permanently.
# Remote File Inclusion (RFI) security rules
# Note: Only whitelist your additional domains or files if needed - do not whitelist hacker domains or files
RewriteCond %{QUERY_STRING} ^.*(http|https|ftp)(%3A|:)(%2F|/)(%2F|/)(w){0,3}.?(blogger|picasa|blogspot|tsunami|petapolitik|photobucket|imgur|imageshack|wordpress\.com|img\.youtube|tinypic\.com|upload\.wikimedia|kkc|start-thegame).*$ [NC,OR]
RewriteCond %{THE_REQUEST} ^.*(http|https|ftp)(%3A|:)(%2F|/)(%2F|/)(w){0,3}.?(blogger|picasa|blogspot|tsunami|petapolitik|photobucket|imgur|imageshack|wordpress\.com|img\.youtube|tinypic\.com|upload\.wikimedia|kkc|start-thegame).*$ [NC]
RewriteRule .* index.php [F]
# 
# Example: Whitelist additional misc files: (example\.php|another-file\.php|phpthumb\.php|thumb\.php|thumbs\.php)
RewriteCond %{REQUEST_URI} (timthumb\.php|phpthumb\.php|thumb\.php|thumbs\.php) [NC]
# Example: Whitelist additional website domains: RewriteCond %{HTTP_REFERER} ^.*(YourWebsite.com|AnotherWebsite.com).*
RewriteCond %{HTTP_REFERER} ^.*example.com.* [OR]
RewriteCond %{HTTP_USER_AGENT} ^(.*Googlebot.*|.*otherbot.*|.*anotherbot.*) [NC]
RewriteRule . - [S=1]

Security Log Ignore logging approach…

Go to the BPS Security Log page, enter Googlebot for the Add User Agents/Bots to Ignore/Not Log and click the Add/Ignore button.

Summary/Analysis

The problem starts with code in a theme or plugin doing something in a way that triggers a 403 error.

Here are the pitfalls of whitelisting the googlebot. Hackers can spoof/fake that they are the googlebot. So if you whitelist the googlebot you may actually be whitelisting a hacking attempt against your website using timthumb as the exploit.

Personally I think it is better just to ignore the nuisance errors because this means your site is still protected, but you are no longer seeing log entries due to the original problem – poor coding work in a Theme or plugin.

What negates the pitfall of ignoring actual real googlebot errors is this – if your site actually does have an image retrieval problem then Google will let you know about it in Google Webmaster Tools.

Something legitimate is actually being blocked or a legitimate bot/User Agent is being blocked because HEAD Requests are blocked

If you would like to allow a Bot to make HEAD Requests on your website then modify this Request Method filter in your Root .htaccess file and add the name of the Bot that you want to allow to make a HEAD Request on your website.  Example Bot:  UptimeRobot/1.0.  You would add the Bot name with Regex .* to match the entire User Agent string and not add the version number of the Bot to this HTTP_USER_AGENT whitelist condition (RewriteCond):

RewriteCond %{HTTP_USER_AGENT} !^(.*UptimeRobot.*) [NC]

# REQUEST METHODS FILTERED
# If you want to allow HEAD Requests use BPS Custom Code and 
# remove/delete HEAD| from the Request Method filter.
# Example: RewriteCond %{REQUEST_METHOD} ^(TRACE|DELETE|TRACK|DEBUG) [NC]
# The TRACE, DELETE, TRACK and DEBUG Request methods should never be removed.
RewriteCond %{REQUEST_METHOD} ^(HEAD|TRACE|DELETE|TRACK|DEBUG) [NC]
RewriteCond %{HTTP_USER_AGENT} !^(.*UptimeRobot.*|.*bot1.*|.*bot2.*|.*bot3.*) [NC]
RewriteRule ^(.*)$ - [F]

Another approach is to remove HEAD from the nuisance security filter.

BPS Pro 11.6+ & BPS free .53.2+
You may see this code or the 11.5+/.53.1+ code in your root htaccess file.  The code does the same exact thing and is whitelisted in the same exact way.

# REQUEST METHODS FILTERED
# If you want to allow HEAD Requests use BPS Custom Code and copy
# this entire REQUEST METHODS FILTERED section of code to this BPS Custom Code
# text box: CUSTOM CODE REQUEST METHODS FILTERED.
# See the CUSTOM CODE REQUEST METHODS FILTERED help text for additional steps.
RewriteCond %{REQUEST_METHOD} ^(TRACE|DELETE|TRACK|DEBUG) [NC]
RewriteRule ^(.*)$ - [F]
#RewriteCond %{REQUEST_METHOD} ^(HEAD) [NC]
#RewriteRule ^(.*)$ /wp-content/plugins/bulletproof-security/405.php [L]

BPS Pro 11.5+ & BPS free .53.1+

# REQUEST METHODS FILTERED
# If you want to allow HEAD Requests use BPS Custom Code and copy
# this entire REQUEST METHODS FILTERED section of code to this BPS Custom Code
# text box: CUSTOM CODE REQUEST METHODS FILTERED.
# See the CUSTOM CODE REQUEST METHODS FILTERED help text for additional steps.
RewriteCond %{REQUEST_METHOD} ^(TRACE|DELETE|TRACK|DEBUG) [NC]
RewriteRule ^(.*)$ - [F]
#RewriteCond %{REQUEST_METHOD} ^(HEAD) [NC]
#RewriteRule ^(.*)$ - [R=405,L]

BPS Pro 11.4|BPS free .53 and lower versions

# REQUEST METHODS FILTERED
# If you want to allow HEAD Requests use BPS Custom Code and 
# remove/delete HEAD| from the Request Method filter.
# Example: RewriteCond %{REQUEST_METHOD} ^(TRACE|DELETE|TRACK|DEBUG) [NC]
# The TRACE, DELETE, TRACK and DEBUG Request methods should never be removed.
RewriteCond %{REQUEST_METHOD} ^(TRACE|DELETE|TRACK|DEBUG) [NC]
RewriteRule ^(.*)$ - [F]

[AITpro Admin]

A newer approach using the new Custom Code text boxes would be to…

1. Copy this REQUEST METHODS FILTERED .htaccess code to this BPS Custom Code text box: CUSTOM CODE REQUEST METHODS FILTERED
2. Click the Save Root Custom Code button.
3. Go to the BPS Security Modes page and click the Root Folder BulletProof Mode Activate button.
BPS Pro 11.6+ & BPS free .53.2+
You may see this code or the 11.5+/.53.1+ code in your root htaccess file.  The code does the same exact thing and is whitelisted in the same exact way.

# REQUEST METHODS FILTERED
# If you want to allow HEAD Requests use BPS Custom Code and copy
# this entire REQUEST METHODS FILTERED section of code to this BPS Custom Code
# text box: CUSTOM CODE REQUEST METHODS FILTERED.
# See the CUSTOM CODE REQUEST METHODS FILTERED help text for additional steps.
RewriteCond %{REQUEST_METHOD} ^(TRACE|DELETE|TRACK|DEBUG) [NC]
RewriteRule ^(.*)$ - [F]
#RewriteCond %{REQUEST_METHOD} ^(HEAD) [NC]
#RewriteRule ^(.*)$ /wp-content/plugins/bulletproof-security/405.php [L]

BPS Pro 11.5+ & BPS free .53.1+

# REQUEST METHODS FILTERED
# If you want to allow HEAD Requests use BPS Custom Code and copy
# this entire REQUEST METHODS FILTERED section of code to this BPS Custom Code
# text box: CUSTOM CODE REQUEST METHODS FILTERED.
# See the CUSTOM CODE REQUEST METHODS FILTERED help text for additional steps.
RewriteCond %{REQUEST_METHOD} ^(TRACE|DELETE|TRACK|DEBUG) [NC]
RewriteRule ^(.*)$ - [F]
#RewriteCond %{REQUEST_METHOD} ^(HEAD) [NC]
#RewriteRule ^(.*)$ - [R=405,L]

BPS Pro 11.4|BPS free .53 and lower versions

# REQUEST METHODS FILTERED
# If you want to allow HEAD Requests use BPS Custom Code and 
# remove/delete HEAD| from the Request Method filter.
# Example: RewriteCond %{REQUEST_METHOD} ^(TRACE|DELETE|TRACK|DEBUG) [NC]
# The TRACE, DELETE, TRACK and DEBUG Request methods should never be removed.
RewriteCond %{REQUEST_METHOD} ^(TRACE|DELETE|TRACK|DEBUG) [NC]
RewriteRule ^(.*)$ - [F]

[Kouichi Sugawara]

Hi AITpro

I have Googlebot Nuisance 403 error on Security Log.
There are three type of solutions on your Forum Site.
My server is VPS.

Which one of solution is recommeded as 1st try ?
And Where to be wrote on .htaccess Custom Code, such as Top or HEAD or Bottom ?

[AITpro Admin]

I recommend using this method above:  http://forum.ait-pro.com/forums/topic/googlebot-googlebot-image-403-error/#post-7695.

[Author]

Posts