Security Log – What matters and what does not matter

[Bro Ignatius Mary]

Dozens and dozens of Security Log 403 errors are being logged. I do not understand this. I give some examples below, where the REQUEST_URL is valid. Why the error?

>>>>>>>>>>> 403 GET or Other Request Error Logged - February 28, 2013 - 4:43 pm <<<<<<<<<<<   
REMOTE_ADDR: 184.169.203.101 
Host Name: ec2-184-169-203-101.us-west-1.compute.amazonaws.com 
HTTP_CLIENT_IP: 
HTTP_FORWARDED: 
HTTP_X_FORWARDED_FOR: 10.196.14.144, 10.164.21.113 
HTTP_X_CLUSTER_CLIENT_IP: 
REQUEST_METHOD: GET 
HTTP_REFERER: 
REQUEST_URI: /bubba/900 
QUERY_STRING: 
HTTP_USER_AGENT: UnwindFetchor/1.0 (+http://www.gnip.com/)

And what do these mean? Is someone trying to break in to js files?

>>>>>>>>>>> 403 GET or Other Request Error Logged - February 28, 2013 - 4:26 pm <<<<<<<<<<< 
REMOTE_ADDR: 64.125.188.25 
Host Name: n1.dfw.ces.cvnt.net 
HTTP_CLIENT_IP: 
HTTP_FORWARDED: 
HTTP_X_FORWARDED_FOR: 
HTTP_X_CLUSTER_CLIENT_IP: 
REQUEST_METHOD: GET 
HTTP_REFERER: 
REQUEST_URI: /bubba/wp-content/plugins/bulletproof-security/admin/js/bulletproof-security-admin-2.js?ver=3.5.1 
QUERY_STRING: 
HTTP_USER_AGENT: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.1) Gecko/2008070208 Firefox/3.0.1

[AITpro Admin]

The first log entry is the UnwindFetchor bot making a HEAD Request to your website.  If you would like to allow bots to make a HEAD Request then remove/delete HEAD from the Request Method Filter as shown in this Forum Topic:  http://forum.ait-pro.com/forums/topic/mailchimp-tracking-code-causing-403/#post-1640  .  Or if you choose not to allow this then you will see millions of these until the end of time.  You can just disregard them.

The second log entry looks like some sort of recon for the bulletproof security js file.  This is Forbidden by the Plugin Firewall.  The Plugin Firewall forbids any external access to any plugin files, including the BPS Pro plugin files.

The only log entries that need any attention from you would be these in the Security Log Read Me First Forum Topic link below:

http://forum.ait-pro.com/forums/topic/security-log-http-error-log-read-me-first/

This Forum Topic link below explains that most log entries can be ignored and why:

http://forum.ait-pro.com/forums/topic/security-log-security-log-403-errors/#post-1694

 

[David at Popehat]

Uhm… If the log events in question will occur in the “millions… until the end of time” and if they’re so unimportant that “you can just disregard them” and if they are not among the “log entries that need any attention”, then why on earth does the plugin bloat the log with them at all?

Is it not obvious that this will require manual copying and deletion of the rapidly growing log every day or two?

The plugin effectively imposes the following choice:

Either (a) remove the filter, even though you might want it, or (b) keep the filter, but dedicate yourself to a life of manual log file management, since the log will grow like crazy when the filter’s on and will prompt you every day or two with a warning banner on the dashboard, or (c) turn off logging.

Please note that neither (a) nor (b) nor (c) is satisfactory for users who want to retain the filter, retain logging, and have a life not given over to managing a log file filled with unhelpful content.

Wouldn’t it make more sense to write trivial events (if at all) to a secondary log file (rolling, with only the most recent few files retained), to write only the most important events to the main log, and then to have a size monitor with dashboard warnings only for the important log?

Frequent warning banners and calls for manual file management because the huge log is yet again filled with trivial info— let’s just say that this feature seems ill conceived.

[AITpro Admin]

For BPS Pro folks all log files are automatically zipped and emailed based on user preferences/options settings in S-Monitor. There is no manual maintenance required for any BPS Pro log files since the entire system is automated.

For BPS Free folks the next version release of BPS will contain an additional option setting to exclude frequent unimportant/trivial logged events based on the User Agent String.  S-Monitor cannot be added to BPS free due to the complexity of how S-Monitor is tied into BPS Pro – it is the central monitoring and alerting system for BPS Pro.  Trying to include this feature in BPS Free would increase our support times to an unsatisfactory level and have a negative impact on BPS Pro Development and Growth.

[David at Popehat]

For BPS Free folks the next version release of BPS will contain an additional option setting to exclude frequent unimportant/trivial logged events based on User Agent.

That should help. Thanks.

Since Free should induce or persuade users to buy Pro, it seems important for Free to avoid annoyances that make users doubt the design judgment behind Pro.

[AITpro Admin]

The Security/HTTP Error Log should really only be used for troubleshooting possible plugin or coding conflicts/issues/problems and then be turned off.  There is no point in logging all the typical scanning/probing/reconning/sniffing/scraping/hacking that goes on day in and day out.

[David at Popehat]

Agreed. So is it on or off by default?

[AITpro Admin]

I recommend turning it Off.

[David at Popehat]

So do I. But is it on xor off when you first install BPS Free? What is the default setting for logging?

[AITpro Admin]

Oh of course it is set to On for a number of technical reasons and Support reasons.

QASI0378:  BPS Feature Trial Testing Results and Support Impact:

Setting Error Logging to Off by default

Support Impact:  Very Bad.

Severe user confusion resulting in unnecessary support downtime answering questions of this general nature.

My error log is not logging errors, is not working,  is broken, etc.

Setting Error Logging to On by default

Support Impact:  Very Good.

User responses are based on adjusting personal settings and configurations.  It is assumed that error logging is working correctly.

Additional Notes:  When users perform BPS upgrades any customizations to .htaccess code or option settings are not changed and are retained.

[David at Popehat]

You say “Oh of course it is set to On for a number of technical reasons and Support reasons” and you “recommend turning it Off“?

Consider two cases:
(A) telling people which button to click to turn on logging
(B) telling people why their log file keeps filling up and splatting on the dashboard, and explaining to people which 80% of the log’s contents they can safely ignore

I’m surprised that (A) requires more support time than (B). But since you’ve measured the support impact, I guess that’s all there is to say! Except, of course, “Nevermind! I recommend turning it off!” 😉

[AITpro Admin]

I just go by testing results/stats and don’t really have a say or opinion about what the testers and assessors come up with.  😉  They do thorough testing with trial groups and make their assessments.  Me, I just do plain old paint by numbers and copy and paste.  LOL

[AITpro Admin]

I actually checked on this because I was curious about it.  Here is what I was told.  75% of the people looked at their Security Log file and it was blank.  On the testing sheet they marked off that Security logging did not work.  😉  The assessors also give suggestions/recommendations to improve options.  They recommended having a text message displaying “security logging is not turned on – click here to turn it on”.  This sounds like a winner to me so I checked further to see why this was not done and was told this – “if you ask the user to check his/her security log for errors and the error log is turned off then how are you going to troubleshoot the problem”.  Yeah I guess they are on the ball.  😉

[David at Popehat]

Thanks for the follow-up. Honestly, it sounds as if the way to strike the balance is to have the log on by default, but to write only truly noteworthy events to it.

Logging information that can safely be ignored in response to events that happen very often (e.g., harmless bot automations) seems like a good way to cause the log to fill up way too quickly.

Another option– not as effective, since it’s just a band-aid– would be to increase the log file size threshold that causes the warning banner to appear.

[AITpro Admin]

Yeah I thought of that question and asked that too.  The answer:  IP Addresses, hostnames and User Agents can all be faked.  The Google Bot was attempting to hack the AITpro websites for over a week.  😉  The last time I checked Google does not do much hacking.  LOL

So basically what I was told is that since it is not possible to tell if any of these variables/logged events are real or not or if they are a serious threat or not then the only logical option left is to leave it up to the user to decide what they want to do about logged events based on the User Agent.  Hence the new option in the next version of BPS to choose which User Agents to ignore/not log.

The larger the log file the slower the BPS options page will load so that is out as well.  From what I was told 500KB causes a slight page load time increase and 1MB causes a very significant page load time increase.  😉

[Author]

Posts