Outbrain 403 error – Whitelist Outbrain

Home Forums BulletProof Security Free Outbrain 403 error – Whitelist Outbrain

This topic contains 14 replies, has 6 voices, and was last updated by  AITpro Admin 4 years, 6 months ago.

Viewing 15 posts - 1 through 15 (of 15 total)
  • Author
    Posts
  • #1993

    KenB
    Member

    Just in case anyone else encounters this, Outbrain’s crawlers have ‘java’ in the user agent string. That’s blocked by default in BPS. Find the similar two lines and change them to:

    RewriteCond %{HTTP_USER_AGENT} (havij|libwww-perl|wget|python|nikto|curl|scan|winhttp|clshttp|loader) [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} (;|<|>|'|"|\)|\(|%0A|%0D|%22|%27|%28|%3C|%3E|%00).*(libwww-perl|wget|python|nikto|curl|scan|winhttp|HTTrack|clshttp|archiver|loader|email|harvest|extract|grab|miner) [NC,OR]
    #1995

    AITpro Admin
    Keymaster

    Wow thanks!  I have been meaning to figure this one out.  Thanks for the great contribution!

    #2067

    Stacy
    Member

    Hi Ken, I came here from the Outbrain forum because I had initially asked this question over there. I tried to add the code you provided….both in my htaccess file in my cpanel and in the BPS file in my wordpress but neither seemed to help (when I try to add it to the wordpress plugin I immediately get an email saying that BPS has blocked an attack on my website). Outbrain still won’t recognize my url.  Where exactly should I be adding this code? Any insight would be greatly appreciated. Thanks so much for your suggestion so far, I have been ripping my hair out trying to use Outbrain with BPS!  Stacy

    #2069

    Stacy
    Member

    I should also add, the folks at Outbrain gave me a list of CDIR ranges and user agent but I have no idea where to place these. It sounds like either one of these fixes might work but I’m not sure where to plug in the info.

    #2072

    AITpro Admin
    Keymaster

    UPDATED Instructions: 11-25-2014

    1. Copy the BPS Query String Exploits code below (the java user agent has already been removed in the code below) to this BPS Root Custom Code text box: CUSTOM CODE BPSQSE BPS QUERY STRING EXPLOITS: Modify Query String Exploit code here
    2. Click the Save Root Custom Code button.
    3. Go to the BPS Security Modes page and click the Root Folder BulletProof Mode Activate button.

    # BEGIN BPSQSE BPS QUERY STRING EXPLOITS
    # The libwww-perl User Agent is forbidden - Many bad bots use libwww-perl modules, but some good bots use it too.
    # Good sites such as W3C use it for their W3C-LinkChecker. 
    # Use BPS Custom Code to add or remove user agents temporarily or permanently from the 
    # User Agent filters directly below or to modify/edit/change any of the other security code rules below.
    RewriteCond %{HTTP_USER_AGENT} (havij|libwww-perl|wget|python|nikto|curl|scan|winhttp|clshttp|loader) [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} (%0A|%0D|%27|%3C|%3E|%00) [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} (;|<|>|'|"|\)|\(|%0A|%0D|%22|%27|%28|%3C|%3E|%00).*(libwww-perl|wget|python|nikto|curl|scan|winhttp|HTTrack|clshttp|archiver|loader|email|harvest|extract|grab|miner) [NC,OR]
    RewriteCond %{THE_REQUEST} (\?|\*|%2a)+(%20+|\\s+|%20+\\s+|\\s+%20+|\\s+%20+\\s+)(http|https)(:/|/) [NC,OR]
    RewriteCond %{THE_REQUEST} etc/passwd [NC,OR]
    RewriteCond %{THE_REQUEST} cgi-bin [NC,OR]
    RewriteCond %{THE_REQUEST} (%0A|%0D|\\r|\\n) [NC,OR]
    RewriteCond %{REQUEST_URI} owssvr\.dll [NC,OR]
    RewriteCond %{HTTP_REFERER} (%0A|%0D|%27|%3C|%3E|%00) [NC,OR]
    RewriteCond %{HTTP_REFERER} \.opendirviewer\. [NC,OR]
    RewriteCond %{HTTP_REFERER} users\.skynet\.be.* [NC,OR]
    RewriteCond %{QUERY_STRING} [a-zA-Z0-9_]=(http|https):// [NC,OR]
    RewriteCond %{QUERY_STRING} [a-zA-Z0-9_]=(\.\.//?)+ [NC,OR]
    RewriteCond %{QUERY_STRING} [a-zA-Z0-9_]=/([a-z0-9_.]//?)+ [NC,OR]
    RewriteCond %{QUERY_STRING} \=PHP[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12} [NC,OR]
    RewriteCond %{QUERY_STRING} (\.\./|%2e%2e%2f|%2e%2e/|\.\.%2f|%2e\.%2f|%2e\./|\.%2e%2f|\.%2e/) [NC,OR]
    RewriteCond %{QUERY_STRING} ftp\: [NC,OR]
    RewriteCond %{QUERY_STRING} (http|https)\: [NC,OR]
    RewriteCond %{QUERY_STRING} \=\|w\| [NC,OR]
    RewriteCond %{QUERY_STRING} ^(.*)/self/(.*)$ [NC,OR]
    RewriteCond %{QUERY_STRING} ^(.*)cPath=(http|https)://(.*)$ [NC,OR]
    RewriteCond %{QUERY_STRING} (\<|%3C).*script.*(\>|%3E) [NC,OR]
    RewriteCond %{QUERY_STRING} (<|%3C)([^s]*s)+cript.*(>|%3E) [NC,OR]
    RewriteCond %{QUERY_STRING} (\<|%3C).*embed.*(\>|%3E) [NC,OR]
    RewriteCond %{QUERY_STRING} (<|%3C)([^e]*e)+mbed.*(>|%3E) [NC,OR]
    RewriteCond %{QUERY_STRING} (\<|%3C).*object.*(\>|%3E) [NC,OR]
    RewriteCond %{QUERY_STRING} (<|%3C)([^o]*o)+bject.*(>|%3E) [NC,OR]
    RewriteCond %{QUERY_STRING} (\<|%3C).*iframe.*(\>|%3E) [NC,OR]
    RewriteCond %{QUERY_STRING} (<|%3C)([^i]*i)+frame.*(>|%3E) [NC,OR] 
    RewriteCond %{QUERY_STRING} base64_encode.*\(.*\) [NC,OR]
    RewriteCond %{QUERY_STRING} base64_(en|de)code[^(]*\([^)]*\) [NC,OR]
    RewriteCond %{QUERY_STRING} GLOBALS(=|\[|\%[0-9A-Z]{0,2}) [OR]
    RewriteCond %{QUERY_STRING} _REQUEST(=|\[|\%[0-9A-Z]{0,2}) [OR]
    RewriteCond %{QUERY_STRING} ^.*(\(|\)|<|>|%3c|%3e).* [NC,OR]
    RewriteCond %{QUERY_STRING} ^.*(\x00|\x04|\x08|\x0d|\x1b|\x20|\x3c|\x3e|\x7f).* [NC,OR]
    RewriteCond %{QUERY_STRING} (NULL|OUTFILE|LOAD_FILE) [OR]
    RewriteCond %{QUERY_STRING} (\.{1,}/)+(motd|etc|bin) [NC,OR]
    RewriteCond %{QUERY_STRING} (localhost|loopback|127\.0\.0\.1) [NC,OR]
    RewriteCond %{QUERY_STRING} (<|>|'|%0A|%0D|%27|%3C|%3E|%00) [NC,OR]
    RewriteCond %{QUERY_STRING} concat[^\(]*\( [NC,OR]
    RewriteCond %{QUERY_STRING} union([^s]*s)+elect [NC,OR]
    RewriteCond %{QUERY_STRING} union([^a]*a)+ll([^s]*s)+elect [NC,OR]
    RewriteCond %{QUERY_STRING} \-[sdcr].*(allow_url_include|allow_url_fopen|safe_mode|disable_functions|auto_prepend_file) [NC,OR]
    RewriteCond %{QUERY_STRING} (;|<|>|'|"|\)|%0A|%0D|%22|%27|%3C|%3E|%00).*(/\*|union|select|insert|drop|delete|update|cast|create|char|convert|alter|declare|order|script|set|md5|benchmark|encode) [NC,OR]
    RewriteCond %{QUERY_STRING} (sp_executesql) [NC]
    RewriteRule ^(.*)$ - [F]
    # END BPSQSE BPS QUERY STRING EXPLOITS
    #10856

    AITpro Admin
    Keymaster

    OLDER INFO: See the UPDATED Instructions in the previous post reply above

    During a BPS plugin upgrade customizations that you have done to your .htaccess code are not changed, but you can save your modified code to BPS Custom Code to save it permanently.

    1. Copy your modified BPS Query String Exploits code from your root .htaccess file to this BPS Custom Code text box:  CUSTOM CODE BPSQSE BPS QUERY STRING EXPLOITS: Modify Query String Exploit code here 
    2. Click the Save Root Custom Code button.
    3. Go to the BPS Security Modes page and click the Root Folder BulletProof Mode Activate button.
    Note: Copy all the BPS Query string code starting from # BEGIN BPSQSE BPS QUERY STRING EXPLOITS to # END BPSQSE BPS QUERY STRING EXPLOITS.

    # BEGIN BPSQSE BPS QUERY STRING EXPLOITS
    ...
    ...
    ...
    # END BPSQSE BPS QUERY STRING EXPLOITS
    #17236

    Greg
    Participant

    [Forum Topic has been merged into this relavent Topic]

    Hi

    I’m trying to whitelist a crawler. I have the following information but no idea how to do this in the htaccess file:
    You will need to whitelist the following crawler IP’s:

    4.28.73.128/27    [4.28.73.128    -> 4.28.73.159]
    74.201.198.64/27   [74.201.198.64   -> 74.201.198.95]
    204.145.89.32/27  [204.145.89.32  -> 204.145.89.63]
    64.94.43.96/27    [64.94.43.96    -> 64.94.43.127]
    74.217.148.96/27  [74.217.148.96  -> 74.217.148.127]
    4.71.33.160/27    [4.71.33.160    -> 4.71.33.191]
    50.31.185.32/27   [50.31.185.32   -> 50.31.185.63]
    64.74.232.32/27   [64.74.232.32   -> 64.74.232.63]
    
    and also our user-agent:  "Mozilla/5.0 (Java) outbrain

    If anyone can please show me how to whitelist this in BPS .htaccess without breaking any of the security it would be much appreciated – I don’t know much about .htaccess but I can copy and paste!

    Thanks

    #17239

    AITpro Admin
    Keymaster

    @ Greg – see the forum topic info above for the solution to whitelist/allow the Outbrain user agent.

    #19356

    Chazz
    Participant

    [Topic has been merged into this relevant Topic]

    i am using an external service that is being blocked from accessing the site. the company gave me their ip ranges and user agent, but i think i am adding them in the wrong place (page-specific instead of sitewide). where do i add the ip ranges and user agent so they are whitelisted for the entire site?

    thanx!

    #19360

    AITpro Admin
    Keymaster

    [Topic has been merged into this relevant Topic]

    What is the name of the external service?  Post a BPS Security Log entry for what is being blocked.

    #19364

    Chazz
    Participant

    [Topic has been merged into this relevant Topic]

    the external service is outbrain http://www.outbrain.com/

    my log hasn’t been updating, perhaps i need to chmod the log file/folder? is there a specific box in the custom code page to add sitewide whitelisted ip ranges/user agents?

    #19375

    AITpro Admin
    Keymaster

    @ Chazz – see the forum topic info above for the solution to whitelist/allow the Outbrain user agent.

    #19381

    Chazz
    Participant

    i think that did it, thanx again for the amazing support!

    #21737

    Joshua
    Participant

    [Topic has been merged into this relevant Topic]

    Hi all, I need help with whitelisting crawler for outbrain.

    Outbrain.com is a content syndication service. They recommend your content on other websites. Before they do, they have their crawlers go through their website.

    That’s where the problems start for me. I can’t progress with outbrain without having the crawlers look at my site.

    I emailed them asking, and they just pointed me to their FAQ page, step 3, whitelist their crawlers.
    http://help.outbrain.com/customer/portal/articles/1461551?b_id=1524

    I have a feeling is it BP security that is blocking the crawlers. So, how do I do whitelist them? I understand it’s something to do with editing the secure htaccess file. But which one, and how?

    I’ve never had a problem with crawlers from search engines like Google or even Facebook crawlers. So I’m pretty surprised that I have a problem with this.

    Thanks for any and all help!

    #21742

    AITpro Admin
    Keymaster

    @ Joshua – your Topic has been merged into this relevant Topic.  The solution is in this Topic here:  http://forum.ait-pro.com/forums/topic/outbrain-doesnt-work-how-to-fix/#post-2072

Viewing 15 posts - 1 through 15 (of 15 total)

You must be logged in to reply to this topic.