SEMrush Scan – 403 error

Home Forums BulletProof Security Pro SEMrush Scan – 403 error

Tagged: ,

This topic contains 4 replies, has 3 voices, and was last updated by  AITpro Admin 2 years, 8 months ago.

Viewing 5 posts - 1 through 5 (of 5 total)
  • Author
    Posts
  • #25600

    Stefan
    Participant

    Hi,

    i am using Semrush and the Crawling-Tool is blocked.

    [403 GET / HEAD Request: 6. Oktober 2015 - 18:35]
    Event Code: BFHS - Blocked/Forbidden Hacker or Spammer
    Solution: N/A - Hacker/Spammer Blocked/Forbidden
    REMOTE_ADDR: 85.14.248.165
    Host Name: al121.albit.fastwebserver.de
    SERVER_PROTOCOL: HTTP/1.1
    HTTP_CLIENT_IP:
    HTTP_FORWARDED:
    HTTP_X_FORWARDED_FOR:
    HTTP_X_CLUSTER_CLIENT_IP:
    REQUEST_METHOD: GET
    HTTP_REFERER:
    REQUEST_URI: /robots.txt
    QUERY_STRING:
    HTTP_USER_AGENT: Mozilla/5.0 (compatible; seoscanners.net/1; +spider@seoscanners.net)

    I did have test it with in custom code, but it did not fix my problem. Semrush is always blocked.
    How can i whitelist them? Please give me the full custom code block.

    # REQUEST METHODS FILTERED
    # If you want to allow HEAD Requests use BPS Custom Code and
    # remove/delete HEAD| from the Request Method filter.
    # Example: RewriteCond %{REQUEST_METHOD} ^(TRACE|DELETE|TRACK|DEBUG) [NC]
    # The TRACE, DELETE, TRACK and DEBUG Request methods should never be removed.
    RewriteCond %{REQUEST_METHOD} ^(HEAD|TRACE|DELETE|TRACK|DEBUG) [NC] ->> here also tried without HEAD
    RewriteCond %{REMOTE_ADDR} !^(85.14.248.165) [NC] ->> here also tried without this line
    RewriteCond %{HTTP_USER_AGENT} !^(seoscanners.net) [NC] ->> here also tried seoscanners.net/1
    RewriteRule ^(.*)$ - [F]  ->> here also tried [F,L]
    #25604

    AITpro Admin
    Keymaster

    The BPS Query String Exploits User Agent nuisance filters are blocking the word “scan” in the SEMrush User Agent.

    1.  Copy the modified BPS Query String Exploits code below (“scan” has been removed from the code below) to this BPS Root Custom Code text box:  CUSTOM CODE BPSQSE BPS QUERY STRING EXPLOITS: Modify Query String Exploit code here 
    2.  Click the Save Root Custom Code button.
    3. Go to the BPS Security Modes page and click the Root Folder BulletProof Mode Activate button.

    # BEGIN BPSQSE BPS QUERY STRING EXPLOITS
    # The libwww-perl User Agent is forbidden - Many bad bots use libwww-perl modules, but some good bots use it too.
    # Good sites such as W3C use it for their W3C-LinkChecker. 
    # Use BPS Custom Code to add or remove user agents temporarily or permanently from the 
    # User Agent filters directly below or to modify/edit/change any of the other security code rules below.
    RewriteCond %{HTTP_USER_AGENT} (havij|libwww-perl|wget|python|nikto|curl|java|winhttp|clshttp|loader) [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} (%0A|%0D|%27|%3C|%3E|%00) [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} (;|<|>|'|"|\)|\(|%0A|%0D|%22|%27|%28|%3C|%3E|%00).*(libwww-perl|wget|python|nikto|curl|java|winhttp|HTTrack|clshttp|archiver|loader|email|harvest|extract|grab|miner) [NC,OR]
    RewriteCond %{THE_REQUEST} (\?|\*|%2a)+(%20+|\\s+|%20+\\s+|\\s+%20+|\\s+%20+\\s+)(http|https)(:/|/) [NC,OR]
    RewriteCond %{THE_REQUEST} etc/passwd [NC,OR]
    RewriteCond %{THE_REQUEST} cgi-bin [NC,OR]
    RewriteCond %{THE_REQUEST} (%0A|%0D|\\r|\\n) [NC,OR]
    RewriteCond %{REQUEST_URI} owssvr\.dll [NC,OR]
    RewriteCond %{HTTP_REFERER} (%0A|%0D|%27|%3C|%3E|%00) [NC,OR]
    RewriteCond %{HTTP_REFERER} \.opendirviewer\. [NC,OR]
    RewriteCond %{HTTP_REFERER} users\.skynet\.be.* [NC,OR]
    RewriteCond %{QUERY_STRING} [a-zA-Z0-9_]=(http|https):// [NC,OR]
    RewriteCond %{QUERY_STRING} [a-zA-Z0-9_]=(\.\.//?)+ [NC,OR]
    RewriteCond %{QUERY_STRING} [a-zA-Z0-9_]=/([a-z0-9_.]//?)+ [NC,OR]
    RewriteCond %{QUERY_STRING} \=PHP[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12} [NC,OR]
    RewriteCond %{QUERY_STRING} (\.\./|%2e%2e%2f|%2e%2e/|\.\.%2f|%2e\.%2f|%2e\./|\.%2e%2f|\.%2e/) [NC,OR]
    RewriteCond %{QUERY_STRING} ftp\: [NC,OR]
    RewriteCond %{QUERY_STRING} (http|https)\: [NC,OR] 
    RewriteCond %{QUERY_STRING} \=\|w\| [NC,OR]
    RewriteCond %{QUERY_STRING} ^(.*)/self/(.*)$ [NC,OR]
    RewriteCond %{QUERY_STRING} ^(.*)cPath=(http|https)://(.*)$ [NC,OR]
    RewriteCond %{QUERY_STRING} (\<|%3C).*script.*(\>|%3E) [NC,OR]
    RewriteCond %{QUERY_STRING} (<|%3C)([^s]*s)+cript.*(>|%3E) [NC,OR]
    RewriteCond %{QUERY_STRING} (\<|%3C).*embed.*(\>|%3E) [NC,OR]
    RewriteCond %{QUERY_STRING} (<|%3C)([^e]*e)+mbed.*(>|%3E) [NC,OR]
    RewriteCond %{QUERY_STRING} (\<|%3C).*object.*(\>|%3E) [NC,OR]
    RewriteCond %{QUERY_STRING} (<|%3C)([^o]*o)+bject.*(>|%3E) [NC,OR]
    RewriteCond %{QUERY_STRING} (\<|%3C).*iframe.*(\>|%3E) [NC,OR]
    RewriteCond %{QUERY_STRING} (<|%3C)([^i]*i)+frame.*(>|%3E) [NC,OR] 
    RewriteCond %{QUERY_STRING} base64_encode.*\(.*\) [NC,OR]
    RewriteCond %{QUERY_STRING} base64_(en|de)code[^(]*\([^)]*\) [NC,OR]
    RewriteCond %{QUERY_STRING} GLOBALS(=|\[|\%[0-9A-Z]{0,2}) [OR]
    RewriteCond %{QUERY_STRING} _REQUEST(=|\[|\%[0-9A-Z]{0,2}) [OR]
    RewriteCond %{QUERY_STRING} ^.*(\(|\)|<|>|%3c|%3e).* [NC,OR]
    RewriteCond %{QUERY_STRING} ^.*(\x00|\x04|\x08|\x0d|\x1b|\x20|\x3c|\x3e|\x7f).* [NC,OR]
    RewriteCond %{QUERY_STRING} (NULL|OUTFILE|LOAD_FILE) [OR]
    RewriteCond %{QUERY_STRING} (\.{1,}/)+(motd|etc|bin) [NC,OR]
    RewriteCond %{QUERY_STRING} (localhost|loopback|127\.0\.0\.1) [NC,OR]
    RewriteCond %{QUERY_STRING} (<|>|'|%0A|%0D|%27|%3C|%3E|%00) [NC,OR]
    RewriteCond %{QUERY_STRING} concat[^\(]*\( [NC,OR]
    RewriteCond %{QUERY_STRING} union([^s]*s)+elect [NC,OR]
    RewriteCond %{QUERY_STRING} union([^a]*a)+ll([^s]*s)+elect [NC,OR]
    RewriteCond %{QUERY_STRING} \-[sdcr].*(allow_url_include|allow_url_fopen|safe_mode|disable_functions|auto_prepend_file) [NC,OR]
    RewriteCond %{QUERY_STRING} (;|<|>|'|"|\)|%0A|%0D|%22|%27|%3C|%3E|%00).*(/\*|union|select|insert|drop|delete|update|cast|create|char|convert|alter|declare|order|script|set|md5|benchmark|encode) [NC,OR]
    RewriteCond %{QUERY_STRING} (sp_executesql) [NC]
    RewriteRule ^(.*)$ - [F]
    # END BPSQSE BPS QUERY STRING EXPLOITS
    #25607

    Stefan
    Participant

    Thank you, now it works!

    #26526

    Sebastian Schertel
    Participant

    I will have to add to this topic since I want to completely BLOCK seoscanners.net because I asked SEMrush about it. Here is their reply:

    Hi Sebastian,

    Thank you for contacting SEMrush! The exact name of our Bot is SemrushBot-SA. You may also seem SemrushBot–Desktop or SemrushBot-Mobile. We are in no way affiliated with seoscanners.net.

    I hope this helps and please let me know if there is anything else I can help you with!

    So they do claim that seoscanners.net has nothing to do with them. How can I block it completly from accessing my site, please? Thanks. 🙂

    #26529

    AITpro Admin
    Keymaster

    The seoscanners bot/User Agent is already blocked by default.  This forum topic is about how to NOT block the seoscanners bot/User Agent.  So you would not have to do anything, unless you want to allow/whitelist this bot/User Agent.

Viewing 5 posts - 1 through 5 (of 5 total)

You must be logged in to reply to this topic.