Website Headers Check Tool – Head Request 500 Error

[Johnny]

Hi, I have another query if you don’t mind. So I’m using the Broken Links Checker Plugin, and so I followed the steps to add the required code like so

# REQUEST METHODS FILTERED
# If you want to allow HEAD Requests use BPS Custom Code and copy
# this entire REQUEST METHODS FILTERED section of code to this BPS Custom Code
# text box: CUSTOM CODE REQUEST METHODS FILTERED.
# See the CUSTOM CODE REQUEST METHODS FILTERED help text for additional steps.
RewriteCond %{REQUEST_METHOD} ^(TRACE|DELETE|TRACK|DEBUG) [NC]
RewriteRule ^(.*)$ - [F]
#RewriteCond %{REQUEST_METHOD} ^(HEAD) [NC]
#RewriteRule ^(.*)$ /wp-content/plugins/bulletproof-security/405.php [L]

Now, using the “Website Headers Check Tool” from System Info, I check both GET and HEAD methods for accessing the headers information of my homepage, and both are working. Now if I remove that .htaccess code, I expected to get a 405 Error when trying to use the HEAD method to retrieve the headers – but instead I get a 500 internal server error, which is not logged in by the Security Log. I tried to enable wp_debug and the associated log, and check the error log when I run the Check Headers feature, but it will not record anything (well it will record something different – I’m using a plugin to force ssl, so it’s just recording the fact that in current WordPress versions another command is preferred to the command that the plugin is using to force ssl), so I have no idea how to check where the 500 internal server error is coming from when requesting the headers via HEAD method. So my first question is should I get a 405 error? And if so, what could be causing the 500 internal server error instead?

I run Sucuri + Wordfence + W3TC with Cloudflare caching and Flexible SSL. I disabled the wp-content blocking hardening feature of Sucuri. Another question I have is that currently if I try to access www.example.com/wp-content (for example) it works, but I just get a blank page. Is this normal or should I receive an error of not having permission? And final question, does enabling the HEAD method for the functionality of Broken Link Checker pose a security risk that should be taken into account? Thanks!

[AITpro Admin]

Yep, we are aware of the problem with the HEAD Request checking tool.  The problem is with the BPS HEAD Request checking tool code itself.  Note: The 500 Error only occurs on some sites and not other sites. So that is what needs to be figured out and changed. It is on the scheduled task list of things to fix in BPS .54.6.  500 Internal Server Errors are not logged in the BPS Security Log and would only be logged in your Server’s log files.

Yep, you should see a blank page and not an error when going to URL:  example.com/wp-content.

Nope, allowing HEAD Requests does not reduce your website security.  The REQUEST METHODS FILTERED htaccess code is intended to block bad bots, which make HEAD Requests from hitting a website.  So it falls under nuisance prevention and not website security.

[AITpro Admin]

I did some testing and figured out why the 500 error occurs, but not why it only occurs on some sites and not other sites.

Last year we created significant new htaccess code:  https://forum.ait-pro.com/forums/topic/root-and-wp-admin-htaccess-file-significant-changes/

This new htaccess code is actually much better for a lot of reasons, but does not work on some older servers.  So we dumbed the code down to the code you posted above.

# REQUEST METHODS FILTERED
# If you want to allow HEAD Requests use BPS Custom Code and copy
# this entire REQUEST METHODS FILTERED section of code to this BPS Custom Code
# text box: CUSTOM CODE REQUEST METHODS FILTERED.
# See the CUSTOM CODE REQUEST METHODS FILTERED help text for additional steps.
RewriteCond %{REQUEST_METHOD} ^(TRACE|DELETE|TRACK|DEBUG) [NC]
RewriteRule ^(.*)$ - [F]
RewriteCond %{REQUEST_METHOD} ^(HEAD) [NC]
RewriteRule ^(.*)$ - [R=405,L]

On one test site I see the 500 errors, but everything works fine on another test site. I have no idea why that is. What I did figure out is the cause of the 500 error on the site where the 500 error occurs. 😉

This code is redirect code that redirects 405 HTTP Status errors to this file: ErrorDocument 405 /wp-content/plugins/bulletproof-security/405.php

This code rewrites HEAD Requests to the 405 HEAD Request logging template: RewriteRule ^(.*)$ /wp-content/plugins/bulletproof-security/405.php [L]

This code redirects HEAD Requests as a 405 HTTP Status error: RewriteRule ^(.*)$ - [R=405,L], which is then handled by the ErrorDocument 405 /wp-content/plugins/bulletproof-security/405.php code to handle the redirect to the 405 Security Logging template.

When using the HEAD Request tool what happens is an infinite redirect loop error problem occurs, which results in a 500 error since the ErrorDocument redirect code and the 405 RewriteRule cause a redirect loop situation, but only on some sites??? Fortunately in all other regular scenarios (when actual 405 errors are occuring) besides when using the HEAD Request checking tool the 405 RewriteRule works perfectly fine. Note: The cURL Request checking code returns a Header Response using: CURLOPT_HEADER and CURLOPT_NOBODY, which I believe is an important factor in the problem scenario.

So in summary there is actually not a problem with the BPS code and is just an odd scenario that occurs on some sites. You can try the much better optimum code above and see if it works on your particular server. That would solve the 500 error problem. If it works then add that better code to BPS Custom Code.

[Johnny]

Thank you very much for your responses, and my apologies for delayed answer, I was asleep. Okay, I’ve tried the new code and it works in the sense that it doesn’t give the 500 Error and gives the regular 405 error upon using the Headers checking tool. However, when I use it, I get the warning:

“Broken Link Checker plugin HEAD Request Method filter problem detected.”

My question is if the HEAD method is blocked in this manner, would the Broken Link Checker not use any of the other methods for retrieving headers info? And if no, does this mean that I should switch back to the previous code?

[AITpro Admin]

You would still need to comment out these 2 lines of code with # signs just like the other code to allow and not block HEAD Requests.  You would only uncomment this code if you wanted to check your Headers with the Website Headers Check Tool.  Both versions of the code do the same exact thing > Block HEAD Requests.

#RewriteCond %{REQUEST_METHOD} ^(HEAD) [NC]
#RewriteRule ^(.*)$ - [R=405,L]

[Johnny]

Okay, I see, thank you very much!

[Author]

Posts