how to block bots ?

Home Forums BulletProof Security Free how to block bots ?

Viewing 9 posts - 1 through 9 (of 9 total)
  • Author
    Posts
  • #15498
    Cedric B
    Participant

    Hello,

    is there a way to block specific bots with bulleproof security ?
    thanks

    #15500
    AITpro Admin
    Keymaster

    Blocking by specific bot names or IP addresses can be a very time consuming and ongoing thing to do and if you have 100’s of lines of .htaccess code with blocked IP addresses and bot names then this can cause your website to perform slower than allowing the bots to visit your site.

    It is better to block by “bad action” vs creating massive lists of bots or IP address to block.  BPS takes an Action approach to security:  bad bot/hacker/spammer/other jackass does bad action X and Y is the result = Blocked/Forbidden.

    If you are still interested in blocking specific bots then you can add that list/code of bad bots that you want to block to BPS Custom Code in this Root Custom Code text box:  CUSTOM CODE BOTTOM HOTLINKING/FORBID COMMENT SPAMMERS/BLOCK BOTS/BLOCK IP/REDIRECT CODE: Add miscellaneous code here

    After adding custom code to a BPS Root Custom Code text box, go to the BPS Security Modes page and click the Root Folder BulletProof Mode Activate button.

    If you are looking for a list of bad bots then I believe you will find a list on the PerishablePress website.  Here is the section of code that allows you to add bad bots/User Agents.

    http://perishablepress.com/5g-blacklist-2013/

    # 5G:[USER AGENTS]
    <IfModule mod_setenvif.c>
    # SetEnvIfNoCase User-Agent ^$ keep_out
    SetEnvIfNoCase User-Agent (binlar|casper|cmsworldmap|comodo|diavol|dotbot|feedfinder|flicky|ia_archiver|jakarta|kmccrew|nutch|planetwork|purebot|pycurl|skygrid|sucker|turnit|vikspider|zmeu) keep_out
    <limit GET POST PUT>
    Order Allow,Deny
    Allow from all
    Deny from env=keep_out
    </limit>
    </IfModule>
    #15523
    Cedric B
    Participant

    Actually i want to block bots that are looking for your backlinks, such as  MJ12bot and AhrefsBot

    so if i understand well, i have to put this code below in the area CUSTOM CODE BOTTOM HOTLINKING/FORBID COMMENT SPAMMERS/BLOCK BOTS/BLOCK IP/REDIRECT CODE: Add miscellaneous code here:

    # 5G:[USER AGENTS]
    <IfModule mod_setenvif.c>
    # SetEnvIfNoCase User-Agent ^$ keep_out
    SetEnvIfNoCase User-Agent (MJ12bot|AhrefsBot) keep_out
    <limit GET POST PUT>
    Order Allow,Deny
    Allow from all
    Deny from env=keep_out
    </limit>
    </IfModule>

    Thanks

    #15525
    AITpro Admin
    Keymaster

    Yep, you got it.  That code is correct to block both of those bots.

    #15526
    Cedric B
    Participant

    ok i got it.
    Thanks a lot

    #16014
    Cedric B
    Participant

    Hi again, sorry but this script does not seem to be enough to block bots (ie :ahref, majestic bots, etc..) ….. I wrote down this script in custom file and it’s not working …i think we also have to specify it in robot.txt as well…(but not sure how to do it)

    Thanks anyway

    #16016
    AITpro Admin
    Keymaster

    The robots.txt file is where you would add instructions/code for good bots.  Bad bots ignore any instructions/code you add in a robots.txt file.  Did you do all of the Custom Code steps?  After saving your custom code you need to go to the Security Modes page and click the Root folder BulletProof Mode Activate button.

    Custom Code video tutorial
    http://forum.ait-pro.com/video-tutorials/#custom-code

    #16089
    Cedric B
    Participant

    yes, i did all of these steps, and the code appears in the “current root htaccess file ”  in “htaccess file editor” tab but i’ve recently sent around 500 links to my site, and ahref can see them i can send you my login details if you want ..

    thanks

    #16090
    AITpro Admin
    Keymaster

    Hmm not really sure what to tell you.  We do not bother with trying to block individual bots sitewide since the task of adding bad bots would go on forever.  We just use this Server Protocol HTTP/1.0 bot blocking code below, which blocks the majority of bad bots sniffing around the Login page since that is a page you want to protect from bad bots.

    http://forum.ait-pro.com/forums/topic/protect-login-page-from-brute-force-login-attacks/

    # BRUTE FORCE LOGIN PAGE PROTECTION
    # Protects the Login page from SpamBots, HackerBots & Proxies
    # that use Server Protocol HTTP/1.0 or a blank User Agent
    RewriteCond %{REQUEST_URI} ^(/wp-login\.php|.*wp-login\.php.*)$
    RewriteCond %{HTTP_USER_AGENT} ^$ [OR]
    RewriteCond %{THE_REQUEST} HTTP/1\.0$ [OR]
    RewriteCond %{SERVER_PROTOCOL} HTTP/1\.0$
    RewriteRule ^(.*)$ - [F,L]
Viewing 9 posts - 1 through 9 (of 9 total)
  • You must be logged in to reply to this topic.