Block Referer Spammers – Semalt, Kambasoft, Ranksonic, Buttons-for-website

[AITpro Admin]

The order of custom .htaccess code or Bonus Custom Code does not matter in the CUSTOM CODE BOTTOM HOTLINKING/FORBID COMMENT SPAMMERS/BLOCK BOTS/BLOCK IP/REDIRECT CODE text box.  The only Custom Code text box that the order of custom .htaccess code does matter is the CUSTOM CODE TOP PHP/PHP.INI HANDLER/CACHE CODE text box, which should have custom .htaccess code in this order:  1. php/php.ini handler .htaccess code, 2. the X-Content-Type-Options nosniff Bonus Custom code and then 3. htaccess caching code in this Custom Code text box.

[Jeff Rivett]

I’ve been testing the ‘Referral Exclusion List’ feature in Google Analytics, and it seems to keep referral spam from cluttering up my site stats. Is there any particular advantage in using the BPS custom code instead?

[AITpro Admin]

There is no negative impact to a website that I have noticed from Referer Spammer bots besides just skewing Metrics so you do not need to add the BPS Bonus Custom code to block those bot requests from hitting your site(s).  I see a growing trend towards the Referer bots sending fake tracking calls directly to your Google Analytics Tracking ID so at some point the BPS Bonus Custom code may not be useful at all and only filtering out Referer spam Metrics in GA or other Metrics apps would be the only way to get rid of that junk traffic.

[Joe]

You can think that I’m crazy, but I like these Ranksonic guys. Never had problems with them, they try to make the world better. Well, anyway mine.

[AITpro Admin]

@Joe – I don’t think people are complaining about the services that Ranksonic offers.  They are complaining about how the Ranksonic bot traffic is not counted as legitimate visitor traffic and causes their Bounce rate to increase.  So basically this has either no positive benefit for their website or has a negative impact for their website.  If the Ranksonic bot was the only bot doing this type of Bounce visit then it would not be a big deal at all, but let’s say you have 100 different bots doing Bounce visits a few times a month.  That would be 300 Bounce visits per month that is not counted as legitimate visitor traffic and that increases a website’s Bounce rate.  300 Bounce visits per month is not that big deal.  1,000 Bounce visitors per month would probably start to cause a negative impact.

I see other bots making visits that are being counted as legitimate traffic so if Ranksonic changes they way their bot visits a website so that it does not increase the Bounce rate then I will remove them from this forum topic/code.

[Roger MacRae]

Is it necessary to use the code as your have written it or can I do it like they do it on http://www.theedesign.com/blog/2015/blocking-spam-referral-traffic-google-analytics.

so instead of:

# Block/Forbid Referer Spammers/Referer Phishing
RewriteCond %{HTTP_REFERER} ^.*(ranksonic\.|semalt\.|kambasoft\.|buttons-for-website\.|buttons-for-your-website\.com|4webmasters\.org|social-buttons\.com|best-seo-offer\.com|best-seo-solution\.com).*$ [NC]
RewriteRule ^(.*)$ - [F]

Can I use:

SetEnvIfNoCase Referer semalt.com spambot=yes
SetEnvIfNoCase Referer darodar.com spambot=yes
SetEnvIfNoCase Referer buttons-for-website.com spambot=yes
SetEnvIfNoCase Referer fbdownloader.com spambot=yes

Order allow,deny
Allow from all
Deny from env=spambot

If I can use either one, is there any advantages for one vs the other?

Also I was wondering if this affects page load speed.

[AITpro Admin]

Yep, you can use either method/code  format.  They do the exact same thing.  Neither code would affect website performance.  Now the most important thing is that a large number of Referer Spammers  and growing every day are now sending fake tracking calls directly to your Google Analytics Tracking ID instead of visiting your website.  So since the Referer Spammer domain is not actually visiting your website then that leaves you with filtering out Referer Spam domains/hostnames in Google Analytics Metrics. The link you posted above has instructions on how to do that.

[alexb]

Hello,

my site is getting hit by a lot of foreign search engine bots so I want to block them all using the biggest (and frequently updated) list I could find so far: http://www.allthingsdemocrat.com/block-bad-bots-in-htaccess.txt

My question: Which of the custom .htaccess code boxes should I paste this into? I’d remove their security htaccess code, I just want to block the bots/crawlers.

Or maybe…is there a way to just block everything except google, yahoo and bing (and humans of course)?

Thanks

[AITpro Admin]

1.  Copy the your bot blocking htaccess code to this BPS Root Custom Code text box:  CUSTOM CODE BOTTOM HOTLINKING/FORBID COMMENT SPAMMERS/BLOCK BOTS/BLOCK IP/REDIRECT CODE: Add miscellaneous code here 
2.  Click the Save Root Custom Code button.
3. Go to the BPS Security Modes page and click the Root Folder BulletProof Mode Activate button.

[alexb]

Thanks, works!  But is there a way to block everything except humans, google yahoo and bing? Or is that not so easy?  Because IP ranges and user agents aren’t set in stone…

[AITpro Admin]

Reverse the equation and security method.  It is always better/more effective to “allow” vs “block”.  Why?  Because “allow” is known and finite.  “Block” is unknown and infinite (may not be infinite in all cases).  For what you are trying to do specifically with blocking bots it is ok to do that, but if you reverse the equation and add a CAPTCHA to all Forms then a human will add the CAPTCHA and all bots will not because they auto-populate Forms and they will be blocked.  Not all CAPTCHA plugins are effective.  BPS Pro JTC Anti-Spam|Anti-Hacker CAPTCHA is 100% effective at blocking 100% of all auto-registering, auto-login and auto-posting bots.  So since it is not really that important to block bots that are hitting your site in general, but it is important to block bots that are trying to automatically use GET or POST on your website Forms then if you install and use a CAPTCHA plugin you will be blocking only bots and not humans for all of your Forms.  I cannot recommend a free CAPTCHA plugin since we use BPS Pro JTC Anti-Spam|Anti-Hacker for our CAPTCHA.

[AITpro Admin]

Oops looks like you have BPS Pro so just enable JTC Anti-Spam|Anti-Hacker, which will block 100% of all bots on your Forms.

[alexb]

Thanks for the reply, I’m already using the JTC captcha, so not sure why you gave me the captcha rundown. I’m not looking for captcha-plugins or even captchas themselves, as search engine spiders/bots are hitting my site en masse.

Maybe I haven’t expressed myself well, I have BPS pro and the security side covered, but that doesn’t help with these crawlers. They don’t care about any kind of forms and aren’t trying to auto-populate anything, they just hammer the server by crawling thousands of pages. My CPU is constantly at 50% and even though I have a high bandwidth allowance, they just waste a ton of resources.

Since I only want to focus on Google, Yahoo and Bing, I want to block all the others. I’m sure a captcha won’t make a difference in that case?

So is there a way to just allow those 3 and automatically block all others, or not?

[AITpro Admin]

Hmm bots have no effect whatsoever on resource usage on our sites so that is odd that they would be causing any significant resource usage on your site/server.  Are you sure that bots are causing the high resource usage issue and not something else?  Since you probably want to allow any/all visitors to your website then only allowing literally google.com, etc. would block any/all visitors to your website besides just literally google.com, etc.  I think you need to figure out exactly what is causing high resource usage first before doing anything else.

Since you have a VPS, Dedicated or you own in-house server then try doing Google searches using search terms like this:  how to reduce server resource usage, how to reduce server memory usage, etc.

[alexb]

I have several sites that are particularly large, as in 10.000 posts and more, so there is quite some fodder for search engines. In the access logs I have a ton of them, googlebot fetches a new page every few seconds and so does bing, yahoo only a few hundred per day but yandex and baidu are all over my access logs, too.

Since I’m only looking for US traffic all those foreign search engines are pretty useless to me, yet eat up valuable resources. But those big ones are already covered in that htaccess bot blocking code I posted earlier, I was just wondering if there was a way to only allow google and humans because at the end of the day I don’t want to have to check my access logs every day and see if there are any new offenders I might have to add. But looks like that’s the route I have to take.

Since implementing that code on the site about 2 hours ago, visits per hour have already been cut in half so looks like it’s working!

[Author]

Posts