Outbrain 403 error – Whitelist Outbrain

[KenB]

Just in case anyone else encounters this, Outbrain’s crawlers have ‘java’ in the user agent string. That’s blocked by default in BPS. Find the similar two lines and change them to:

RewriteCond %{HTTP_USER_AGENT} (havij|libwww-perl|wget|python|nikto|curl|scan|winhttp|clshttp|loader) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} (;|<|>|'|"|\)|\(|%0A|%0D|%22|%27|%28|%3C|%3E|%00).*(libwww-perl|wget|python|nikto|curl|scan|winhttp|HTTrack|clshttp|archiver|loader|email|harvest|extract|grab|miner) [NC,OR]

[AITpro Admin]

Wow thanks!  I have been meaning to figure this one out.  Thanks for the great contribution!

[Stacy]

Hi Ken, I came here from the Outbrain forum because I had initially asked this question over there. I tried to add the code you provided….both in my htaccess file in my cpanel and in the BPS file in my wordpress but neither seemed to help (when I try to add it to the wordpress plugin I immediately get an email saying that BPS has blocked an attack on my website). Outbrain still won’t recognize my url.  Where exactly should I be adding this code? Any insight would be greatly appreciated. Thanks so much for your suggestion so far, I have been ripping my hair out trying to use Outbrain with BPS!  Stacy

[Stacy]

I should also add, the folks at Outbrain gave me a list of CDIR ranges and user agent but I have no idea where to place these. It sounds like either one of these fixes might work but I’m not sure where to plug in the info.

[AITpro Admin]

UPDATED Instructions: 11-25-2014

1. Copy the BPS Query String Exploits code below (the java user agent has already been removed in the code below) to this BPS Root Custom Code text box: CUSTOM CODE BPSQSE BPS QUERY STRING EXPLOITS: Modify Query String Exploit code here
2. Click the Save Root Custom Code button.
3. Go to the BPS Security Modes page and click the Root Folder BulletProof Mode Activate button.

# BEGIN BPSQSE BPS QUERY STRING EXPLOITS
# The libwww-perl User Agent is forbidden - Many bad bots use libwww-perl modules, but some good bots use it too.
# Good sites such as W3C use it for their W3C-LinkChecker. 
# Use BPS Custom Code to add or remove user agents temporarily or permanently from the 
# User Agent filters directly below or to modify/edit/change any of the other security code rules below.
RewriteCond %{HTTP_USER_AGENT} (havij|libwww-perl|wget|python|nikto|curl|scan|winhttp|clshttp|loader) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} (%0A|%0D|%27|%3C|%3E|%00) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} (;|<|>|'|"|\)|\(|%0A|%0D|%22|%27|%28|%3C|%3E|%00).*(libwww-perl|wget|python|nikto|curl|scan|winhttp|HTTrack|clshttp|archiver|loader|email|harvest|extract|grab|miner) [NC,OR]
RewriteCond %{THE_REQUEST} (\?|\*|%2a)+(%20+|\\s+|%20+\\s+|\\s+%20+|\\s+%20+\\s+)(http|https)(:/|/) [NC,OR]
RewriteCond %{THE_REQUEST} etc/passwd [NC,OR]
RewriteCond %{THE_REQUEST} cgi-bin [NC,OR]
RewriteCond %{THE_REQUEST} (%0A|%0D|\\r|\\n) [NC,OR]
RewriteCond %{REQUEST_URI} owssvr\.dll [NC,OR]
RewriteCond %{HTTP_REFERER} (%0A|%0D|%27|%3C|%3E|%00) [NC,OR]
RewriteCond %{HTTP_REFERER} \.opendirviewer\. [NC,OR]
RewriteCond %{HTTP_REFERER} users\.skynet\.be.* [NC,OR]
RewriteCond %{QUERY_STRING} [a-zA-Z0-9_]=(http|https):// [NC,OR]
RewriteCond %{QUERY_STRING} [a-zA-Z0-9_]=(\.\.//?)+ [NC,OR]
RewriteCond %{QUERY_STRING} [a-zA-Z0-9_]=/([a-z0-9_.]//?)+ [NC,OR]
RewriteCond %{QUERY_STRING} \=PHP[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12} [NC,OR]
RewriteCond %{QUERY_STRING} (\.\./|%2e%2e%2f|%2e%2e/|\.\.%2f|%2e\.%2f|%2e\./|\.%2e%2f|\.%2e/) [NC,OR]
RewriteCond %{QUERY_STRING} ftp\: [NC,OR]
RewriteCond %{QUERY_STRING} (http|https)\: [NC,OR]
RewriteCond %{QUERY_STRING} \=\|w\| [NC,OR]
RewriteCond %{QUERY_STRING} ^(.*)/self/(.*)$ [NC,OR]
RewriteCond %{QUERY_STRING} ^(.*)cPath=(http|https)://(.*)$ [NC,OR]
RewriteCond %{QUERY_STRING} (\<|%3C).*script.*(\>|%3E) [NC,OR]
RewriteCond %{QUERY_STRING} (<|%3C)([^s]*s)+cript.*(>|%3E) [NC,OR]
RewriteCond %{QUERY_STRING} (\<|%3C).*embed.*(\>|%3E) [NC,OR]
RewriteCond %{QUERY_STRING} (<|%3C)([^e]*e)+mbed.*(>|%3E) [NC,OR]
RewriteCond %{QUERY_STRING} (\<|%3C).*object.*(\>|%3E) [NC,OR]
RewriteCond %{QUERY_STRING} (<|%3C)([^o]*o)+bject.*(>|%3E) [NC,OR]
RewriteCond %{QUERY_STRING} (\<|%3C).*iframe.*(\>|%3E) [NC,OR]
RewriteCond %{QUERY_STRING} (<|%3C)([^i]*i)+frame.*(>|%3E) [NC,OR] 
RewriteCond %{QUERY_STRING} base64_encode.*\(.*\) [NC,OR]
RewriteCond %{QUERY_STRING} base64_(en|de)code[^(]*\([^)]*\) [NC,OR]
RewriteCond %{QUERY_STRING} GLOBALS(=|\[|\%[0-9A-Z]{0,2}) [OR]
RewriteCond %{QUERY_STRING} _REQUEST(=|\[|\%[0-9A-Z]{0,2}) [OR]
RewriteCond %{QUERY_STRING} ^.*(\(|\)|<|>|%3c|%3e).* [NC,OR]
RewriteCond %{QUERY_STRING} ^.*(\x00|\x04|\x08|\x0d|\x1b|\x20|\x3c|\x3e|\x7f).* [NC,OR]
RewriteCond %{QUERY_STRING} (NULL|OUTFILE|LOAD_FILE) [OR]
RewriteCond %{QUERY_STRING} (\.{1,}/)+(motd|etc|bin) [NC,OR]
RewriteCond %{QUERY_STRING} (localhost|loopback|127\.0\.0\.1) [NC,OR]
RewriteCond %{QUERY_STRING} (<|>|'|%0A|%0D|%27|%3C|%3E|%00) [NC,OR]
RewriteCond %{QUERY_STRING} concat[^\(]*\( [NC,OR]
RewriteCond %{QUERY_STRING} union([^s]*s)+elect [NC,OR]
RewriteCond %{QUERY_STRING} union([^a]*a)+ll([^s]*s)+elect [NC,OR]
RewriteCond %{QUERY_STRING} \-[sdcr].*(allow_url_include|allow_url_fopen|safe_mode|disable_functions|auto_prepend_file) [NC,OR]
RewriteCond %{QUERY_STRING} (;|<|>|'|"|\)|%0A|%0D|%22|%27|%3C|%3E|%00).*(/\*|union|select|insert|drop|delete|update|cast|create|char|convert|alter|declare|order|script|set|md5|benchmark|encode) [NC,OR]
RewriteCond %{QUERY_STRING} (sp_executesql) [NC]
RewriteRule ^(.*)$ - [F]
# END BPSQSE BPS QUERY STRING EXPLOITS

[AITpro Admin]

OLDER INFO: See the UPDATED Instructions in the previous post reply above

During a BPS plugin upgrade customizations that you have done to your .htaccess code are not changed, but you can save your modified code to BPS Custom Code to save it permanently.

1. Copy your modified BPS Query String Exploits code from your root .htaccess file to this BPS Custom Code text box:  CUSTOM CODE BPSQSE BPS QUERY STRING EXPLOITS: Modify Query String Exploit code here 
2. Click the Save Root Custom Code button.
3. Go to the BPS Security Modes page and click the Root Folder BulletProof Mode Activate button.
Note: Copy all the BPS Query string code starting from # BEGIN BPSQSE BPS QUERY STRING EXPLOITS to # END BPSQSE BPS QUERY STRING EXPLOITS.

# BEGIN BPSQSE BPS QUERY STRING EXPLOITS
...
...
...
# END BPSQSE BPS QUERY STRING EXPLOITS

[Greg]

[Forum Topic has been merged into this relavent Topic]

Hi

I’m trying to whitelist a crawler. I have the following information but no idea how to do this in the htaccess file:
You will need to whitelist the following crawler IP’s:

4.28.73.128/27    [4.28.73.128    -> 4.28.73.159]
74.201.198.64/27   [74.201.198.64   -> 74.201.198.95]
204.145.89.32/27  [204.145.89.32  -> 204.145.89.63]
64.94.43.96/27    [64.94.43.96    -> 64.94.43.127]
74.217.148.96/27  [74.217.148.96  -> 74.217.148.127]
4.71.33.160/27    [4.71.33.160    -> 4.71.33.191]
50.31.185.32/27   [50.31.185.32   -> 50.31.185.63]
64.74.232.32/27   [64.74.232.32   -> 64.74.232.63]

and also our user-agent:  "Mozilla/5.0 (Java) outbrain

If anyone can please show me how to whitelist this in BPS .htaccess without breaking any of the security it would be much appreciated – I don’t know much about .htaccess but I can copy and paste!

Thanks

[AITpro Admin]

@ Greg – see the forum topic info above for the solution to whitelist/allow the Outbrain user agent.

[Chazz]

[Topic has been merged into this relevant Topic]

i am using an external service that is being blocked from accessing the site. the company gave me their ip ranges and user agent, but i think i am adding them in the wrong place (page-specific instead of sitewide). where do i add the ip ranges and user agent so they are whitelisted for the entire site?

thanx!

[AITpro Admin]

[Topic has been merged into this relevant Topic]

What is the name of the external service?  Post a BPS Security Log entry for what is being blocked.

[Chazz]

[Topic has been merged into this relevant Topic]

the external service is outbrain http://www.outbrain.com/

my log hasn’t been updating, perhaps i need to chmod the log file/folder? is there a specific box in the custom code page to add sitewide whitelisted ip ranges/user agents?

[AITpro Admin]

@ Chazz – see the forum topic info above for the solution to whitelist/allow the Outbrain user agent.

[Chazz]

i think that did it, thanx again for the amazing support!

[Joshua]

[Topic has been merged into this relevant Topic]

Hi all, I need help with whitelisting crawler for outbrain.

Outbrain.com is a content syndication service. They recommend your content on other websites. Before they do, they have their crawlers go through their website.

That’s where the problems start for me. I can’t progress with outbrain without having the crawlers look at my site.

I emailed them asking, and they just pointed me to their FAQ page, step 3, whitelist their crawlers.
http://help.outbrain.com/customer/portal/articles/1461551?b_id=1524

I have a feeling is it BP security that is blocking the crawlers. So, how do I do whitelist them? I understand it’s something to do with editing the secure htaccess file. But which one, and how?

I’ve never had a problem with crawlers from search engines like Google or even Facebook crawlers. So I’m pretty surprised that I have a problem with this.

Thanks for any and all help!

[AITpro Admin]

@ Joshua – your Topic has been merged into this relevant Topic.  The solution is in this Topic here:  http://forum.ait-pro.com/forums/topic/outbrain-doesnt-work-how-to-fix/#post-2072

[Author]

Posts