facebook Bot – facebook externalhit_uatext.php, facebook Bot 403 Error

[AITpro Admin]

This is a nuisance issue and will be taken care of in BPS .48.  What is happening is that something in the facebook externalhit_uatext.php script is causing 403 Errors to be logged.  The facebook bot – facebookexternalhit is NOT being blocked and image retrieval is working correctly.  See the solution below.

>>>>>>>>>>> 403 GET or Other Request Error Logged - February 8, 2013 - 12:04 pm <<<<<<<<<<<
REMOTE_ADDR: 69.171.247.113
Host Name: 69.171.247.113
HTTP_CLIENT_IP:
HTTP_FORWARDED:
HTTP_X_FORWARDED_FOR:
HTTP_X_CLUSTER_CLIENT_IP:
REQUEST_METHOD: GET
HTTP_REFERER:
REQUEST_URI: /aitpro-blog/wp-content/themes/AITpro/images/aitpro-logo-footer.png
QUERY_STRING:
HTTP_USER_AGENT: facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)

Solution:  modify the /bulletproof-security/403.php file to include a check not to log 403 Errors from the facebook bot.

if ($_SERVER['REQUEST_METHOD'] != 'POST' && !preg_match('/facebookexternalhit(.*)/s', $_SERVER['HTTP_USER_AGENT'], $matches) ) {
...
...
...

[AITpro Admin]

The new solution is to go to the BPS Security Log page and add the facebook bot to Add User Agents/Bots to Ignore/Not Log

You can enter facebookexternalhit or facebook or externalhit_uatext to Ignore/Not Log the facebook User Agent/Bot.

Security Log page Blue Read Me help button info

Ignoring/Not Logging User Agents/Bots – Allowing/Logging User Agents/Bots
Adding or Removing User Agents/Bots adds or removes User Agents/Bots to your Database and also writes new code to the 403.php Security Logging template. The 403.php Security Logging file is where the check occurs whether or not to log or not log a User Agent/Bot. It would be foolish and costly to website performance to have your WordPress database handle the task/function/burden of checking which User Agents/Bots to log or not log. WordPress database queries are the most resource draining function of a WordPress website. The more database queries that are happening at the same time on your website the slower your website will perform and load. For this reason the Security Logging check is done from code in the 403.php Security Logging file.

If a particular User Agent/Bot is being logged excessively in your Security Log file you can Ignore/Not Log that particular User Agent/Bot based on the HTTP_USER_AGENT string in your Security Log.

Example User Agent strings:

Mozilla/5.0 (compatible; 008/0.85; http://www.80legs.com/webcrawler.html) 
Gecko/2008032620 facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)

You could enter 008 or 80legs or webcrawler to Ignore/Not Log the 80legs User Agent/Bot. You could enter facebookexternalhit or facebook or externalhit_uatext to Ignore/Not Log the facebook User Agent/Bot.

[Jose]

After I added Hotlink Protection .htaccess code, my website images are not being shown at facebook pages when some user pastes a link to my site.

I added facebook to the whitelist like this.

SetEnvIfNoCase Referer "^(http|https)://www\.mysite\.com.*$" whitelist
SetEnvIfNoCase Referer "^(http|https)://.*google.*" whitelist
SetEnvIfNoCase Referer "^(http|https)://.*yahoo.*" whitelist
SetEnvIfNoCase Referer "^(http|https)://.*bing.*" whitelist
SetEnvIfNoCase Referer "^(http|https)://.*anothersite.wordpress.*" whitelist
SetEnvIfNoCase Referer "^(http|https)://.*facebook.*" whitelist

The images are not being shown at facebook and I receive this kind of security log entries:

[403 GET / HEAD Request: August 22, 2014 4:27 pm]
Event Code: BFHS - Blocked/Forbidden Hacker or Spammer
Solution: N/A - Hacker/Spammer Blocked/Forbidden
REMOTE_ADDR: 173.252.120.118
Host Name: 173.252.120.118
SERVER_PROTOCOL: HTTP/1.1
HTTP_CLIENT_IP:
HTTP_FORWARDED:
HTTP_X_FORWARDED_FOR:
HTTP_X_CLUSTER_CLIENT_IP:
REQUEST_METHOD: GET
HTTP_REFERER:
REQUEST_URI: /wp-content/uploads/2012/10/photo-2645.jpg
QUERY_STRING:
HTTP_USER_AGENT: facebookexternalhit/1.0 (+http://www.facebook.com/externalhit_uatext.php)

[AITpro Admin]

Double check that your actual website domain name is correct in this line of hotlink protection code.

SetEnvIfNoCase Referer "^(http|https)://www\.mysite\.com.*$" whitelist

Double check that you have done all of the BPS Custom Code steps to actually add/create this new code change/modification in your root .htaccess file.
http://forum.ait-pro.com/forums/topic/hotlink-protection-do-not-block-google-bing-or-yahoo/

[Jose]

I double checked and my actual website domain is correct.

The only failure is on facebook pages. Google, Bing, Yahoo, and other site are showing my site images correctly, so I think it should be a Facebook problem. Before I pasted this code it didn’t happen and the linked images performed well on Facebook.

[AITpro Admin]

Are you sure the image links are correct?  Have you checked the links to make sure they are valid links to image files?

Since facebook.com is not shown in the Security Log entry as the Referer then probably the whitelist rule for facebook.com is not doing anything. Try adding the facebook.com IP address as shown below.

SetEnvIfNoCase Referer "^(http|https)://www\.your-website-name\.com.*$" whitelist
SetEnvIfNoCase Referer "^(http|https)://.*google.*" whitelist
SetEnvIfNoCase Referer "^(http|https)://.*yahoo.*" whitelist
SetEnvIfNoCase Referer "^(http|https)://.*bing.*" whitelist

<FilesMatch "\.(gif|jpg|jpeg|png|bmp)$">
Order Allow,Deny
Allow from env=whitelist
# facebook.com IP address
Allow from 173.252.120.118
</FilesMatch>

[Jose]

I tried this code

SetEnvIfNoCase Referer "^(http|https)://www\.mysite\.com.*$" whitelist
SetEnvIfNoCase Referer "^(http|https)://.*google.*" whitelist
SetEnvIfNoCase Referer "^(http|https)://.*yahoo.*" whitelist
SetEnvIfNoCase Referer "^(http|https)://.*bing.*" whitelist
SetEnvIfNoCase Referer "^(http|https)://.*othersite.wordpress.*" whitelist

<FilesMatch "\.(gif|jpg|jpeg|png|bmp)$">
Order Allow,Deny
Allow from env=whitelist
# Add Your Server IP Address
# Note: A Server IP address May be Required for 
# everything to work depending on your Host Server
Allow from 87.106.196.193
# facebook.com IP address
Allow from 173.252.120.

because I saw these different endings at facebook ip address

[403 GET / HEAD Request: August 24, 2014 12:02 am]
Event Code: BFHS - Blocked/Forbidden Hacker or Spammer
Solution: N/A - Hacker/Spammer Blocked/Forbidden
REMOTE_ADDR: 173.252.113.114
Host Name: 173.252.113.114
SERVER_PROTOCOL: HTTP/1.1
HTTP_CLIENT_IP: 
HTTP_FORWARDED: 
HTTP_X_FORWARDED_FOR: 
HTTP_X_CLUSTER_CLIENT_IP: 
REQUEST_METHOD: GET
HTTP_REFERER: 
REQUEST_URI: /wp-content/uploads/2014/04/no-Ivan-vila-as-100x70.jpg
QUERY_STRING: 
HTTP_USER_AGENT: facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)

[403 GET / HEAD Request: August 24, 2014 12:02 am]
Event Code: BFHS - Blocked/Forbidden Hacker or Spammer
Solution: N/A - Hacker/Spammer Blocked/Forbidden
REMOTE_ADDR: 173.252.113.118
Host Name: 173.252.113.118
SERVER_PROTOCOL: HTTP/1.1
HTTP_CLIENT_IP: 
HTTP_FORWARDED: 
HTTP_X_FORWARDED_FOR: 
HTTP_X_CLUSTER_CLIENT_IP: 
REQUEST_METHOD: GET
HTTP_REFERER: 
REQUEST_URI: /wp-content/uploads/2014/04/Bet-Miguel-100x70.jpg
QUERY_STRING: 
HTTP_USER_AGENT: facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)

[403 GET / HEAD Request: August 24, 2014 12:02 am]
Event Code: BFHS - Blocked/Forbidden Hacker or Spammer
Solution: N/A - Hacker/Spammer Blocked/Forbidden
REMOTE_ADDR: 173.252.113.114
Host Name: 173.252.113.114
SERVER_PROTOCOL: HTTP/1.1
HTTP_CLIENT_IP: 
HTTP_FORWARDED: 
HTTP_X_FORWARDED_FOR: 
HTTP_X_CLUSTER_CLIENT_IP: 
REQUEST_METHOD: GET
HTTP_REFERER: 
REQUEST_URI: /wp-content/uploads/2014/07/Omar-de-100x70.jpg
QUERY_STRING: 
HTTP_USER_AGENT: facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)

[403 GET / HEAD Request: August 24, 2014 12:02 am]
Event Code: BFHS - Blocked/Forbidden Hacker or Spammer
Solution: N/A - Hacker/Spammer Blocked/Forbidden
REMOTE_ADDR: 173.252.102.117
Host Name: 173.252.102.117
SERVER_PROTOCOL: HTTP/1.1
HTTP_CLIENT_IP: 
HTTP_FORWARDED: 
HTTP_X_FORWARDED_FOR: 
HTTP_X_CLUSTER_CLIENT_IP: 
REQUEST_METHOD: GET
HTTP_REFERER: 
REQUEST_URI: /wp-content/uploads/2014/06/Exit-Caroline-100x70.jpg
QUERY_STRING: 
HTTP_USER_AGENT: facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)

The images are not displayed on Facebook when I insert a post link (not an image link) but if I only add this file

SetEnvIfNoCase Referer "^(http|https)://.*facebook.*" whitelist

I can see the images if I write at my facebook wall or chat a link to the categories tabs of the site. For example mysite.com/sections/section_1
The posts links don’t show images,

[AITpro Admin]

I forgot that these Security Log entries do not mean anything.  What is being blocked has nothing to do with the actual image retrieval.  These are just nuisance log entries that mean nothing and can be ignored.  See the very beginning of this forum topic for more of an explanation.

I believe what is happening is either this is an issue on the facebook site and has nothing to do with your website or what you are expecting to see on facebook is not working because of something you are doing incorrectly on your site or what you are expecting to see on facebook is just not how that works on facebook.  One of these things is probably true.

This does not appear to have anything at all to do with the Hotlink protection code so I cannot offer you any other advice about the issue.  Sorry.

[Jose]

No matter. It’s not really an important stuff because the others whitelisted rules are working properly for google, bing, yahoo and any other site I added. The fact is that before adding the hotlink protection code, every post link pasted on facebook showed some images to select and this is not happening now where it only shows the link and the excerpt text but not any image. I will continue finding out why this happens and if I know the solution I will post it here. Thanks a lot.

[AITpro Admin]

Sounds good.  Dumb of me to not read the forum topic first since those security log entries will always happen and are not related to this issue/problem.  How exactly do you do this on facebook:  The posts links don’t show images.  How would I add a post link?  I will test this on my facebook account and see what happens.

[AITpro Admin]

Hmm ok I posted a link to this forum site and the facebook image selector picked an image.  This is the facebook Source Code below.  So maybe whitelisting fbstatic-a.akamaihd.net or probably just .*akamaihd.net would work.  I will add hotlink protection code in this forum site and see what happens.

<img class="_42xb img" src="https://fbstatic-a.akamaihd.net/rsrc.php/v2/y4/r/-PAXP-deijE.gif" alt="" height="116" width="116" style="background-image: url(https://fbexternal-a.akamaihd.net/safe_image.php?d=AQAx4ZAYBYZ1s_8S&amp;w=116&amp;h=116&amp;url=http%3A%2F%2Fforum.ait-pro.com%2Fwp-content%2Fuploads%2F2013%2F10%2FJTC-Anti-Spam-300x190.png&amp;cfs=1);" />

[AITpro Admin]

I was able to post a post on facebook with and without the hotlink protection code and with and without facebook.com being whitelisted, but out of 10 test posts, image retrieval worked 8 out of 10 times and failed 2 times.  A logical explanation for something like that would be that the image retrieval script has a timeout / time limitation.  Example:  the image retrieval script tries for 3 seconds to get an image file.  If it cannot get that image file in 3 seconds it does not try again.

[Jose]

Ah, Ok so it probably it depends on the load time of the server. I added fbstatic-a.akamaihd.net to the whitelist but at the moment it’s not working and the image retrieval is not being shown. The hotlink protection code is ok, then. You’re the man. Thank you.

[George Mohan]

How can i white list Facebook Referrer to my website

[Thu Jun 16 19:39:27.627950 2016] [access_compat:error] [pid 399266] [client 141.101.104.108:23058] AH01797: client denied by server configuration: /home/XXXX/public_html/wp-content/plugins/bulletproof-security/403.php, referer: http://XXXXX.com/avanthika/
[Thu Jun 16 19:39:27.177256 2016] [access_compat:error] [pid 399266] [client 141.101.104.108:23058] AH01797: client denied by server configuration: /home/XXXX/public_html/favicon.ico, referer: http://XXXXXX.com/avanthika/
[Thu Jun 16 19:39:26.330340 2016] [access_compat:error] [pid 399257] [client 141.101.104.29:19859] AH01797: client denied by server configuration: /home/XXXX/public_html/wp-content/plugins/bulletproof-security/403.php, referer: https://www.facebook.com/
[Thu Jun 16 19:39:26.330189 2016] [access_compat:error] [pid 399257] [client 141.101.104.29:19859] AH01797: client denied by server configuration: /home/XXXX/public_html/XXXXX, referer: https://m.facebook.com/

[AITpro Admin]

I don’t think you need to whitelist the facebook bot since nothing important is being blocked and you can just ignore the facebook bot user agent so that these log entries are not logged.  If you want to check whether or not BPS is blocking something the facebook bot is doing then do these steps below and recheck your server log file to see if something else on your server is blocking the facebook bot instead of BPS. BPS logs all 403 errors using the /bulletproof-security/403.php logging template file whether or not the 403 errors are caused by BPS.

1. Deactivate Root Folder BulletProof Mode.
2. Turn Off BPS Security Logging.
3. Check your server log file for any more facebook bot log entries.

[Author]

Posts