facebook Bot – facebook externalhit_uatext.php, facebook Bot 403 Error

Home Forums BulletProof Security Free facebook Bot – facebook externalhit_uatext.php, facebook Bot 403 Error

This topic contains 14 replies, has 3 voices, and was last updated by  AITpro Admin 3 years, 4 months ago.

Viewing 15 posts - 1 through 15 (of 15 total)
  • Author
    Posts
  • #2013

    AITpro Admin
    Keymaster

    This is a nuisance issue and will be taken care of in BPS .48.  What is happening is that something in the facebook externalhit_uatext.php script is causing 403 Errors to be logged.  The facebook bot – facebookexternalhit is NOT being blocked and image retrieval is working correctly.  See the solution below.

    >>>>>>>>>>> 403 GET or Other Request Error Logged - February 8, 2013 - 12:04 pm <<<<<<<<<<<
    REMOTE_ADDR: 69.171.247.113
    Host Name: 69.171.247.113
    HTTP_CLIENT_IP:
    HTTP_FORWARDED:
    HTTP_X_FORWARDED_FOR:
    HTTP_X_CLUSTER_CLIENT_IP:
    REQUEST_METHOD: GET
    HTTP_REFERER:
    REQUEST_URI: /aitpro-blog/wp-content/themes/AITpro/images/aitpro-logo-footer.png
    QUERY_STRING:
    HTTP_USER_AGENT: facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)

    Solution:  modify the /bulletproof-security/403.php file to include a check not to log 403 Errors from the facebook bot.

    if ($_SERVER['REQUEST_METHOD'] != 'POST' && !preg_match('/facebookexternalhit(.*)/s', $_SERVER['HTTP_USER_AGENT'], $matches) ) {
    ...
    ...
    ...
    #5896

    AITpro Admin
    Keymaster

    The new solution is to go to the BPS Security Log page and add the facebook bot to 

    You can enter facebookexternalhit or facebook or externalhit_uatext to Ignore/Not Log the facebook User Agent/Bot.

    Security Log page Blue Read Me help button info

    Ignoring/Not Logging User Agents/Bots – Allowing/Logging User Agents/Bots
    Adding or Removing User Agents/Bots adds or removes User Agents/Bots to your Database and also writes new code to the 403.php Security Logging template. The 403.php Security Logging file is where the check occurs whether or not to log or not log a User Agent/Bot. It would be foolish and costly to website performance to have your WordPress database handle the task/function/burden of checking which User Agents/Bots to log or not log. WordPress database queries are the most resource draining function of a WordPress website. The more database queries that are happening at the same time on your website the slower your website will perform and load. For this reason the Security Logging check is done from code in the 403.php Security Logging file.

    If a particular User Agent/Bot is being logged excessively in your Security Log file you can Ignore/Not Log that particular User Agent/Bot based on the HTTP_USER_AGENT string in your Security Log.

    Example User Agent strings:

    Mozilla/5.0 (compatible; 008/0.85; http://www.80legs.com/webcrawler.html) 
    Gecko/2008032620 facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)

    You could enter 008 or 80legs or webcrawler to Ignore/Not Log the 80legs User Agent/Bot. You could enter facebookexternalhit or facebook or externalhit_uatext to Ignore/Not Log the facebook User Agent/Bot.

    #17095

    Jose
    Participant

    After I added Hotlink Protection .htaccess code, my website images are not being shown at facebook pages when some user pastes a link to my site.

    I added facebook to the whitelist like this.

    SetEnvIfNoCase Referer "^(http|https)://www\.mysite\.com.*$" whitelist
    SetEnvIfNoCase Referer "^(http|https)://.*google.*" whitelist
    SetEnvIfNoCase Referer "^(http|https)://.*yahoo.*" whitelist
    SetEnvIfNoCase Referer "^(http|https)://.*bing.*" whitelist
    SetEnvIfNoCase Referer "^(http|https)://.*anothersite.wordpress.*" whitelist
    SetEnvIfNoCase Referer "^(http|https)://.*facebook.*" whitelist

    The images are not being shown at facebook and I receive this kind of security log entries:

    [403 GET / HEAD Request: August 22, 2014 4:27 pm]
    Event Code: BFHS - Blocked/Forbidden Hacker or Spammer
    Solution: N/A - Hacker/Spammer Blocked/Forbidden
    REMOTE_ADDR: 173.252.120.118
    Host Name: 173.252.120.118
    SERVER_PROTOCOL: HTTP/1.1
    HTTP_CLIENT_IP:
    HTTP_FORWARDED:
    HTTP_X_FORWARDED_FOR:
    HTTP_X_CLUSTER_CLIENT_IP:
    REQUEST_METHOD: GET
    HTTP_REFERER:
    REQUEST_URI: /wp-content/uploads/2012/10/photo-2645.jpg
    QUERY_STRING:
    HTTP_USER_AGENT: facebookexternalhit/1.0 (+http://www.facebook.com/externalhit_uatext.php)
    #17104

    AITpro Admin
    Keymaster

    Double check that your actual website domain name is correct in this line of hotlink protection code.

    SetEnvIfNoCase Referer "^(http|https)://www\.mysite\.com.*$" whitelist

    Double check that you have done all of the BPS Custom Code steps to actually add/create this new code change/modification in your root .htaccess file.
    http://forum.ait-pro.com/forums/topic/hotlink-protection-do-not-block-google-bing-or-yahoo/

    #17141

    Jose
    Participant

    I double checked and my actual website domain is correct.

    The only failure is on facebook pages. Google, Bing, Yahoo, and other site are showing my site images correctly, so I think it should be a Facebook problem. Before I pasted this code it didn’t happen and the linked images performed well on Facebook.

    #17142

    AITpro Admin
    Keymaster

    Are you sure the image links are correct?  Have you checked the links to make sure they are valid links to image files?

    Since facebook.com is not shown in the Security Log entry as the Referer then probably the whitelist rule for facebook.com is not doing anything. Try adding the facebook.com IP address as shown below.

    SetEnvIfNoCase Referer "^(http|https)://www\.your-website-name\.com.*$" whitelist
    SetEnvIfNoCase Referer "^(http|https)://.*google.*" whitelist
    SetEnvIfNoCase Referer "^(http|https)://.*yahoo.*" whitelist
    SetEnvIfNoCase Referer "^(http|https)://.*bing.*" whitelist
    
    <FilesMatch "\.(gif|jpg|jpeg|png|bmp)$">
    Order Allow,Deny
    Allow from env=whitelist
    # facebook.com IP address
    Allow from 173.252.120.118
    </FilesMatch>
    #17145

    Jose
    Participant

    I tried this code

    SetEnvIfNoCase Referer "^(http|https)://www\.mysite\.com.*$" whitelist
    SetEnvIfNoCase Referer "^(http|https)://.*google.*" whitelist
    SetEnvIfNoCase Referer "^(http|https)://.*yahoo.*" whitelist
    SetEnvIfNoCase Referer "^(http|https)://.*bing.*" whitelist
    SetEnvIfNoCase Referer "^(http|https)://.*othersite.wordpress.*" whitelist
    
    <FilesMatch "\.(gif|jpg|jpeg|png|bmp)$">
    Order Allow,Deny
    Allow from env=whitelist
    # Add Your Server IP Address
    # Note: A Server IP address May be Required for 
    # everything to work depending on your Host Server
    Allow from 87.106.196.193
    # facebook.com IP address
    Allow from 173.252.120.

    because I saw these different endings at facebook ip address

    [403 GET / HEAD Request: August 24, 2014 12:02 am]
    Event Code: BFHS - Blocked/Forbidden Hacker or Spammer
    Solution: N/A - Hacker/Spammer Blocked/Forbidden
    REMOTE_ADDR: 173.252.113.114
    Host Name: 173.252.113.114
    SERVER_PROTOCOL: HTTP/1.1
    HTTP_CLIENT_IP: 
    HTTP_FORWARDED: 
    HTTP_X_FORWARDED_FOR: 
    HTTP_X_CLUSTER_CLIENT_IP: 
    REQUEST_METHOD: GET
    HTTP_REFERER: 
    REQUEST_URI: /wp-content/uploads/2014/04/no-Ivan-vila-as-100x70.jpg
    QUERY_STRING: 
    HTTP_USER_AGENT: facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)
    
    [403 GET / HEAD Request: August 24, 2014 12:02 am]
    Event Code: BFHS - Blocked/Forbidden Hacker or Spammer
    Solution: N/A - Hacker/Spammer Blocked/Forbidden
    REMOTE_ADDR: 173.252.113.118
    Host Name: 173.252.113.118
    SERVER_PROTOCOL: HTTP/1.1
    HTTP_CLIENT_IP: 
    HTTP_FORWARDED: 
    HTTP_X_FORWARDED_FOR: 
    HTTP_X_CLUSTER_CLIENT_IP: 
    REQUEST_METHOD: GET
    HTTP_REFERER: 
    REQUEST_URI: /wp-content/uploads/2014/04/Bet-Miguel-100x70.jpg
    QUERY_STRING: 
    HTTP_USER_AGENT: facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)
    
    [403 GET / HEAD Request: August 24, 2014 12:02 am]
    Event Code: BFHS - Blocked/Forbidden Hacker or Spammer
    Solution: N/A - Hacker/Spammer Blocked/Forbidden
    REMOTE_ADDR: 173.252.113.114
    Host Name: 173.252.113.114
    SERVER_PROTOCOL: HTTP/1.1
    HTTP_CLIENT_IP: 
    HTTP_FORWARDED: 
    HTTP_X_FORWARDED_FOR: 
    HTTP_X_CLUSTER_CLIENT_IP: 
    REQUEST_METHOD: GET
    HTTP_REFERER: 
    REQUEST_URI: /wp-content/uploads/2014/07/Omar-de-100x70.jpg
    QUERY_STRING: 
    HTTP_USER_AGENT: facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)
    
    [403 GET / HEAD Request: August 24, 2014 12:02 am]
    Event Code: BFHS - Blocked/Forbidden Hacker or Spammer
    Solution: N/A - Hacker/Spammer Blocked/Forbidden
    REMOTE_ADDR: 173.252.102.117
    Host Name: 173.252.102.117
    SERVER_PROTOCOL: HTTP/1.1
    HTTP_CLIENT_IP: 
    HTTP_FORWARDED: 
    HTTP_X_FORWARDED_FOR: 
    HTTP_X_CLUSTER_CLIENT_IP: 
    REQUEST_METHOD: GET
    HTTP_REFERER: 
    REQUEST_URI: /wp-content/uploads/2014/06/Exit-Caroline-100x70.jpg
    QUERY_STRING: 
    HTTP_USER_AGENT: facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)

    The images are not displayed on Facebook when I insert a post link (not an image link) but if I only add this file

    SetEnvIfNoCase Referer "^(http|https)://.*facebook.*" whitelist

    I can see the images if I write at my facebook wall or chat a link to the categories tabs of the site. For example mysite.com/sections/section_1
    The posts links don’t show images,

    #17148

    AITpro Admin
    Keymaster

    I forgot that these Security Log entries do not mean anything.  What is being blocked has nothing to do with the actual image retrieval.  These are just nuisance log entries that mean nothing and can be ignored.  See the very beginning of this forum topic for more of an explanation.

    I believe what is happening is either this is an issue on the facebook site and has nothing to do with your website or what you are expecting to see on facebook is not working because of something you are doing incorrectly on your site or what you are expecting to see on facebook is just not how that works on facebook.  One of these things is probably true.

    This does not appear to have anything at all to do with the Hotlink protection code so I cannot offer you any other advice about the issue.  Sorry.

    #17151

    Jose
    Participant

    No matter. It’s not really an important stuff because the others whitelisted rules are working properly for google, bing, yahoo and any other site I added. The fact is that before adding the hotlink protection code, every post link pasted on facebook showed some images to select and this is not happening now where it only shows the link and the excerpt text but not any image. I will continue finding out why this happens and if I know the solution I will post it here. Thanks a lot.

    #17152

    AITpro Admin
    Keymaster

    Sounds good.  Dumb of me to not read the forum topic first since those security log entries will always happen and are not related to this issue/problem.  How exactly do you do this on facebook:  The posts links don’t show images.  How would I add a post link?  I will test this on my facebook account and see what happens.

    #17153

    AITpro Admin
    Keymaster

    Hmm ok I posted a link to this forum site and the facebook image selector picked an image.  This is the facebook Source Code below.  So maybe whitelisting fbstatic-a.akamaihd.net or probably just .*akamaihd.net would work.  I will add hotlink protection code in this forum site and see what happens.

    <img class="_42xb img" src="https://fbstatic-a.akamaihd.net/rsrc.php/v2/y4/r/-PAXP-deijE.gif" alt="" height="116" width="116" style="background-image: url(https://fbexternal-a.akamaihd.net/safe_image.php?d=AQAx4ZAYBYZ1s_8S&amp;w=116&amp;h=116&amp;url=http%3A%2F%2Fforum.ait-pro.com%2Fwp-content%2Fuploads%2F2013%2F10%2FJTC-Anti-Spam-300x190.png&amp;cfs=1);" />
    #17154

    AITpro Admin
    Keymaster

    I was able to post a post on facebook with and without the hotlink protection code and with and without facebook.com being whitelisted, but out of 10 test posts, image retrieval worked 8 out of 10 times and failed 2 times.  A logical explanation for something like that would be that the image retrieval script has a timeout / time limitation.  Example:  the image retrieval script tries for 3 seconds to get an image file.  If it cannot get that image file in 3 seconds it does not try again.

    #17157

    Jose
    Participant

    Ah, Ok so it probably it depends on the load time of the server. I added fbstatic-a.akamaihd.net to the whitelist but at the moment it’s not working and the image retrieval is not being shown. The hotlink protection code is ok, then. You’re the man. Thank you.

    #29797

    George Mohan
    Participant

    How can i white list Facebook Referrer to my website

    [Thu Jun 16 19:39:27.627950 2016] [access_compat:error] [pid 399266] [client 141.101.104.108:23058] AH01797: client denied by server configuration: /home/XXXX/public_html/wp-content/plugins/bulletproof-security/403.php, referer: http://XXXXX.com/avanthika/
    [Thu Jun 16 19:39:27.177256 2016] [access_compat:error] [pid 399266] [client 141.101.104.108:23058] AH01797: client denied by server configuration: /home/XXXX/public_html/favicon.ico, referer: http://XXXXXX.com/avanthika/
    [Thu Jun 16 19:39:26.330340 2016] [access_compat:error] [pid 399257] [client 141.101.104.29:19859] AH01797: client denied by server configuration: /home/XXXX/public_html/wp-content/plugins/bulletproof-security/403.php, referer: https://www.facebook.com/
    [Thu Jun 16 19:39:26.330189 2016] [access_compat:error] [pid 399257] [client 141.101.104.29:19859] AH01797: client denied by server configuration: /home/XXXX/public_html/XXXXX, referer: https://m.facebook.com/
    
    #29798

    AITpro Admin
    Keymaster

    I don’t think you need to whitelist the facebook bot since nothing important is being blocked and you can just ignore the facebook bot user agent so that these log entries are not logged.  If you want to check whether or not BPS is blocking something the facebook bot is doing then do these steps below and recheck your server log file to see if something else on your server is blocking the facebook bot instead of BPS. BPS logs all 403 errors using the /bulletproof-security/403.php logging template file whether or not the 403 errors are caused by BPS.

    1. Deactivate Root Folder BulletProof Mode.
    2. Turn Off BPS Security Logging.
    3. Check your server log file for any more facebook bot log entries.

Viewing 15 posts - 1 through 15 (of 15 total)

You must be logged in to reply to this topic.