Block Referer Spammers – Semalt, Kambasoft, Ranksonic, Buttons-for-website

Home Forums BulletProof Security Pro Block Referer Spammers – Semalt, Kambasoft, Ranksonic, Buttons-for-website

Viewing 15 posts - 16 through 30 (of 37 total)
  • Author
    Posts
  • #22484
    AITpro Admin
    Keymaster

    The order of custom .htaccess code or Bonus Custom Code does not matter in the CUSTOM CODE BOTTOM HOTLINKING/FORBID COMMENT SPAMMERS/BLOCK BOTS/BLOCK IP/REDIRECT CODE text box.  The only Custom Code text box that the order of custom .htaccess code does matter is the CUSTOM CODE TOP PHP/PHP.INI HANDLER/CACHE CODE text box, which should have custom .htaccess code in this order:  1. php/php.ini handler .htaccess code, 2. the X-Content-Type-Options nosniff Bonus Custom code and then 3. htaccess caching code in this Custom Code text box.

    #23029
    Jeff Rivett
    Participant

    I’ve been testing the ‘Referral Exclusion List’ feature in Google Analytics, and it seems to keep referral spam from cluttering up my site stats. Is there any particular advantage in using the BPS custom code instead?

    #23030
    AITpro Admin
    Keymaster

    There is no negative impact to a website that I have noticed from Referer Spammer bots besides just skewing Metrics so you do not need to add the BPS Bonus Custom code to block those bot requests from hitting your site(s).  I see a growing trend towards the Referer bots sending fake tracking calls directly to your Google Analytics Tracking ID so at some point the BPS Bonus Custom code may not be useful at all and only filtering out Referer spam Metrics in GA or other Metrics apps would be the only way to get rid of that junk traffic.

    #23249
    Joe
    Participant

    You can think that I’m crazy, but I like these Ranksonic guys. Never had problems with them, they try to make the world better. Well, anyway mine.

    #23250
    AITpro Admin
    Keymaster

    @Joe – I don’t think people are complaining about the services that Ranksonic offers.  They are complaining about how the Ranksonic bot traffic is not counted as legitimate visitor traffic and causes their Bounce rate to increase.  So basically this has either no positive benefit for their website or has a negative impact for their website.  If the Ranksonic bot was the only bot doing this type of Bounce visit then it would not be a big deal at all, but let’s say you have 100 different bots doing Bounce visits a few times a month.  That would be 300 Bounce visits per month that is not counted as legitimate visitor traffic and that increases a website’s Bounce rate.  300 Bounce visits per month is not that big deal.  1,000 Bounce visitors per month would probably start to cause a negative impact.

    I see other bots making visits that are being counted as legitimate traffic so if Ranksonic changes they way their bot visits a website so that it does not increase the Bounce rate then I will remove them from this forum topic/code.

    #24392
    Roger MacRae
    Participant

    Is it necessary to use the code as your have written it or can I do it like they do it on http://www.theedesign.com/blog/2015/blocking-spam-referral-traffic-google-analytics.

    so instead of:

    # Block/Forbid Referer Spammers/Referer Phishing
    RewriteCond %{HTTP_REFERER} ^.*(ranksonic\.|semalt\.|kambasoft\.|buttons-for-website\.|buttons-for-your-website\.com|4webmasters\.org|social-buttons\.com|best-seo-offer\.com|best-seo-solution\.com).*$ [NC]
    RewriteRule ^(.*)$ - [F]

    Can I use:

    SetEnvIfNoCase Referer semalt.com spambot=yes
    SetEnvIfNoCase Referer darodar.com spambot=yes
    SetEnvIfNoCase Referer buttons-for-website.com spambot=yes
    SetEnvIfNoCase Referer fbdownloader.com spambot=yes
    
    Order allow,deny
    Allow from all
    Deny from env=spambot

    If I can use either one, is there any advantages for one vs the other?

    Also I was wondering if this affects page load speed.

    #24393
    AITpro Admin
    Keymaster

    Yep, you can use either method/code  format.  They do the exact same thing.  Neither code would affect website performance.  Now the most important thing is that a large number of Referer Spammers  and growing every day are now sending fake tracking calls directly to your Google Analytics Tracking ID instead of visiting your website.  So since the Referer Spammer domain is not actually visiting your website then that leaves you with filtering out Referer Spam domains/hostnames in Google Analytics Metrics. The link you posted above has instructions on how to do that.

    #26568
    alexb
    Participant

    Hello,

    my site is getting hit by a lot of foreign search engine bots so I want to block them all using the biggest (and frequently updated) list I could find so far: http://www.allthingsdemocrat.com/block-bad-bots-in-htaccess.txt

    My question: Which of the custom .htaccess code boxes should I paste this into? I’d remove their security htaccess code, I just want to block the bots/crawlers.

    Or maybe…is there a way to just block everything except google, yahoo and bing (and humans of course)?

    Thanks

    #26569
    AITpro Admin
    Keymaster

    1.  Copy the your bot blocking htaccess code to this BPS Root Custom Code text box:  CUSTOM CODE BOTTOM HOTLINKING/FORBID COMMENT SPAMMERS/BLOCK BOTS/BLOCK IP/REDIRECT CODE: Add miscellaneous code here 
    2.  Click the Save Root Custom Code button.
    3. Go to the BPS Security Modes page and click the Root Folder BulletProof Mode Activate button.

    #26588
    alexb
    Participant

    Thanks, works!  But is there a way to block everything except humans, google yahoo and bing? Or is that not so easy?  Because IP ranges and user agents aren’t set in stone…

    #26592
    AITpro Admin
    Keymaster

    Reverse the equation and security method.  It is always better/more effective to “allow” vs “block”.  Why?  Because “allow” is known and finite.  “Block” is unknown and infinite (may not be infinite in all cases).  For what you are trying to do specifically with blocking bots it is ok to do that, but if you reverse the equation and add a CAPTCHA to all Forms then a human will add the CAPTCHA and all bots will not because they auto-populate Forms and they will be blocked.  Not all CAPTCHA plugins are effective.  BPS Pro JTC Anti-Spam|Anti-Hacker CAPTCHA is 100% effective at blocking 100% of all auto-registering, auto-login and auto-posting bots.  So since it is not really that important to block bots that are hitting your site in general, but it is important to block bots that are trying to automatically use GET or POST on your website Forms then if you install and use a CAPTCHA plugin you will be blocking only bots and not humans for all of your Forms.  I cannot recommend a free CAPTCHA plugin since we use BPS Pro JTC Anti-Spam|Anti-Hacker for our CAPTCHA.

    #26593
    AITpro Admin
    Keymaster

    Oops looks like you have BPS Pro so just enable JTC Anti-Spam|Anti-Hacker, which will block 100% of all bots on your Forms.

    #26595
    alexb
    Participant

    Thanks for the reply, I’m already using the JTC captcha, so not sure why you gave me the captcha rundown. I’m not looking for captcha-plugins or even captchas themselves, as search engine spiders/bots are hitting my site en masse.

    Maybe I haven’t expressed myself well, I have BPS pro and the security side covered, but that doesn’t help with these crawlers. They don’t care about any kind of forms and aren’t trying to auto-populate anything, they just hammer the server by crawling thousands of pages. My CPU is constantly at 50% and even though I have a high bandwidth allowance, they just waste a ton of resources.

    Since I only want to focus on Google, Yahoo and Bing, I want to block all the others. I’m sure a captcha won’t make a difference in that case?

    So is there a way to just allow those 3 and automatically block all others, or not?

    #26598
    AITpro Admin
    Keymaster

    Hmm bots have no effect whatsoever on resource usage on our sites so that is odd that they would be causing any significant resource usage on your site/server.  Are you sure that bots are causing the high resource usage issue and not something else?  Since you probably want to allow any/all visitors to your website then only allowing literally google.com, etc. would block any/all visitors to your website besides just literally google.com, etc.  I think you need to figure out exactly what is causing high resource usage first before doing anything else.

    Since you have a VPS, Dedicated or you own in-house server then try doing Google searches using search terms like this:  how to reduce server resource usage, how to reduce server memory usage, etc.

    #26602
    alexb
    Participant

    I have several sites that are particularly large, as in 10.000 posts and more, so there is quite some fodder for search engines. In the access logs I have a ton of them, googlebot fetches a new page every few seconds and so does bing, yahoo only a few hundred per day but yandex and baidu are all over my access logs, too.

    Since I’m only looking for US traffic all those foreign search engines are pretty useless to me, yet eat up valuable resources. But those big ones are already covered in that htaccess bot blocking code I posted earlier, I was just wondering if there was a way to only allow google and humans because at the end of the day I don’t want to have to check my access logs every day and see if there are any new offenders I might have to add. But looks like that’s the route I have to take.

    Since implementing that code on the site about 2 hours ago, visits per hour have already been cut in half so looks like it’s working!

Viewing 15 posts - 16 through 30 (of 37 total)
  • You must be logged in to reply to this topic.