Google Adwords Instant | www.google.com/adsbot.htm | robots.txt

Home Forums BulletProof Security Pro Google Adwords Instant | www.google.com/adsbot.htm | robots.txt

This topic contains 1 reply, has 2 voices, and was last updated by  AITpro Admin 6 months, 3 weeks ago.

Viewing 2 posts - 1 through 2 (of 2 total)
  • Author
    Posts
  • #34656

    Rafael Da Costa
    Participant

    Hi There,

    I have got few entry like this on my Security log:

    [403 GET Request: 2017-11-28 - 10:58 am] 
    BPS Pro: 13.3.1 WP: 4.9 
    Event Code: BFHS - Blocked/Forbidden Hacker or Spammer 
    Solution: N/A - Hacker/Spammer Blocked/Forbidden 
    REMOTE_ADDR: 66.102.6.163 
    Host Name: google-proxy-66-102-6-163.google.com 
    SERVER_PROTOCOL: HTTP/1.1 
    HTTP_X_FORWARDED_FOR: 66.102.6.163 
    REQUEST_URI: /?keyword=&matchtype=%20https://www.MYURL.COM%20?device=c 
    QUERY_STRING: keyword=&matchtype=%20https://www.MYURL.COM%20?device=c 
    HTTP_USER_AGENT: Google-Adwords-Instant (+http://www.google.com/adsbot.html)

    Would be good to allow this bot? How would I put it on a white-list? What is the best practice for a robots.txt?

    Thanks

     

    #34658

    AITpro Admin
    Keymaster

    3 most likely causes for the problem…

    1. Simulated RFI hacking attempt.
    2. The GET request is actually a HEAD request.
    3. Whitespace URI problem.

    #1. Try this first…
    1. Copy the modified BPS Query String Exploits below to this BPS Root Custom Code text box: CUSTOM CODE BPSQSE BPS QUERY STRING EXPLOITS
    2. Click the Save Root Custom Code button.
    3. Go to the BPS Security Modes page and click the Root Folder BulletProof Mode Activate button.

    # BEGIN BPSQSE BPS QUERY STRING EXPLOITS
    # The libwww-perl User Agent is forbidden - Many bad bots use libwww-perl modules, but some good bots use it too.
    # Good sites such as W3C use it for their W3C-LinkChecker. 
    # Use BPS Custom Code to add or remove user agents temporarily or permanently from the 
    # User Agent filters directly below or to modify/edit/change any of the other security code rules below.
    RewriteCond %{HTTP_USER_AGENT} (havij|libwww-perl|wget|python|nikto|curl|scan|java|winhttp|clshttp|loader) [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} (%0A|%0D|%27|%3C|%3E|%00) [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} (;|<|>|'|"|\)|\(|%0A|%0D|%22|%27|%28|%3C|%3E|%00).*(libwww-perl|wget|python|nikto|curl|scan|java|winhttp|HTTrack|clshttp|archiver|loader|email|harvest|extract|grab|miner) [NC,OR]
    RewriteCond %{THE_REQUEST} (\?|\*|%2a)+(%20+|\\s+|%20+\\s+|\\s+%20+|\\s+%20+\\s+)(http|https)(:/|/) [NC,OR]
    RewriteCond %{THE_REQUEST} etc/passwd [NC,OR]
    RewriteCond %{THE_REQUEST} cgi-bin [NC,OR]
    RewriteCond %{THE_REQUEST} (%0A|%0D|\\r|\\n) [NC,OR]
    RewriteCond %{REQUEST_URI} owssvr\.dll [NC,OR]
    RewriteCond %{HTTP_REFERER} (%0A|%0D|%27|%3C|%3E|%00) [NC,OR]
    RewriteCond %{HTTP_REFERER} \.opendirviewer\. [NC,OR]
    RewriteCond %{HTTP_REFERER} users\.skynet\.be.* [NC,OR]
    #RewriteCond %{QUERY_STRING} [a-zA-Z0-9_]=(http|https):// [NC,OR]
    RewriteCond %{QUERY_STRING} [a-zA-Z0-9_]=(\.\.//?)+ [NC,OR]
    #RewriteCond %{QUERY_STRING} [a-zA-Z0-9_]=/([a-z0-9_.]//?)+ [NC,OR]
    RewriteCond %{QUERY_STRING} \=PHP[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12} [NC,OR]
    RewriteCond %{QUERY_STRING} (\.\./|%2e%2e%2f|%2e%2e/|\.\.%2f|%2e\.%2f|%2e\./|\.%2e%2f|\.%2e/) [NC,OR]
    RewriteCond %{QUERY_STRING} ftp\: [NC,OR]
    #RewriteCond %{QUERY_STRING} (http|https)\: [NC,OR] 
    RewriteCond %{QUERY_STRING} \=\|w\| [NC,OR]
    RewriteCond %{QUERY_STRING} ^(.*)/self/(.*)$ [NC,OR]
    RewriteCond %{QUERY_STRING} ^(.*)cPath=(http|https)://(.*)$ [NC,OR]
    RewriteCond %{QUERY_STRING} (\<|%3C).*script.*(\>|%3E) [NC,OR]
    RewriteCond %{QUERY_STRING} (<|%3C)([^s]*s)+cript.*(>|%3E) [NC,OR]
    RewriteCond %{QUERY_STRING} (\<|%3C).*embed.*(\>|%3E) [NC,OR]
    RewriteCond %{QUERY_STRING} (<|%3C)([^e]*e)+mbed.*(>|%3E) [NC,OR]
    RewriteCond %{QUERY_STRING} (\<|%3C).*object.*(\>|%3E) [NC,OR]
    RewriteCond %{QUERY_STRING} (<|%3C)([^o]*o)+bject.*(>|%3E) [NC,OR]
    RewriteCond %{QUERY_STRING} (\<|%3C).*iframe.*(\>|%3E) [NC,OR]
    RewriteCond %{QUERY_STRING} (<|%3C)([^i]*i)+frame.*(>|%3E) [NC,OR] 
    RewriteCond %{QUERY_STRING} base64_encode.*\(.*\) [NC,OR]
    RewriteCond %{QUERY_STRING} base64_(en|de)code[^(]*\([^)]*\) [NC,OR]
    RewriteCond %{QUERY_STRING} GLOBALS(=|\[|\%[0-9A-Z]{0,2}) [OR]
    RewriteCond %{QUERY_STRING} _REQUEST(=|\[|\%[0-9A-Z]{0,2}) [OR]
    RewriteCond %{QUERY_STRING} ^.*(\(|\)|<|>|%3c|%3e).* [NC,OR]
    RewriteCond %{QUERY_STRING} ^.*(\x00|\x04|\x08|\x0d|\x1b|\x20|\x3c|\x3e|\x7f).* [NC,OR]
    RewriteCond %{QUERY_STRING} (NULL|OUTFILE|LOAD_FILE) [OR]
    RewriteCond %{QUERY_STRING} (\.{1,}/)+(motd|etc|bin) [NC,OR]
    RewriteCond %{QUERY_STRING} (localhost|loopback|127\.0\.0\.1) [NC,OR]
    RewriteCond %{QUERY_STRING} (<|>|'|%0A|%0D|%27|%3C|%3E|%00) [NC,OR]
    RewriteCond %{QUERY_STRING} concat[^\(]*\( [NC,OR]
    RewriteCond %{QUERY_STRING} union([^s]*s)+elect [NC,OR]
    RewriteCond %{QUERY_STRING} union([^a]*a)+ll([^s]*s)+elect [NC,OR]
    RewriteCond %{QUERY_STRING} \-[sdcr].*(allow_url_include|allow_url_fopen|safe_mode|disable_functions|auto_prepend_file) [NC,OR]
    RewriteCond %{QUERY_STRING} (;|<|>|'|"|\)|%0A|%0D|%22|%27|%3C|%3E|%00).*(/\*|union|select|insert|drop|delete|update|cast|create|char|convert|alter|declare|order|script|set|md5|benchmark|encode) [NC,OR]
    RewriteCond %{QUERY_STRING} (sp_executesql) [NC]
    RewriteRule ^(.*)$ - [F]
    # END BPSQSE BPS QUERY STRING EXPLOITS

    #2. Try this next…
    1. Copy the modified REQUEST METHODS FILTERED htaccess code below to this BPS Root Custom Code text box: CUSTOM CODE REQUEST METHODS FILTERED.
    2. Click the Save Root Custom Code button.
    3. Go to the BPS Security Modes page and click the Root Folder BulletProof Mode Activate button.

    # REQUEST METHODS FILTERED
    # If you want to allow HEAD Requests use BPS Custom Code and copy
    # this entire REQUEST METHODS FILTERED section of code to this BPS Custom Code
    # text box: CUSTOM CODE REQUEST METHODS FILTERED.
    # See the CUSTOM CODE REQUEST METHODS FILTERED help text for additional steps.
    RewriteCond %{REQUEST_METHOD} ^(TRACE|DELETE|TRACK|DEBUG) [NC]
    RewriteRule ^(.*)$ - [F]
    #RewriteCond %{REQUEST_METHOD} ^(HEAD) [NC]
    #RewriteRule ^(.*)$ /wp-content/plugins/bulletproof-security/405.php [L]

    #3. Try this last…
    Comment out this security rule with a # sign as shown below:
    See the steps in #1 above for the steps to copy BPS Query String Exploits code to BPS Custom Code.  If you are following the logical order of possible issues/causes of the problem from #1 to #3 then you would have already copied BPS Query String Exploits code to BPS Custom Code in step #1.  😉

    #RewriteCond %{THE_REQUEST} (\?|\*|%2a)+(%20+|\\s+|%20+\\s+|\\s+%20+|\\s+%20+\\s+)(http|https)(:/|/) [NC,OR]
Viewing 2 posts - 1 through 2 (of 2 total)

You must be logged in to reply to this topic.