Failed to open stream: Too many open files

Home Forums BulletProof Security Pro Failed to open stream: Too many open files

Viewing 15 posts - 1 through 15 (of 15 total)
  • Author
    Posts
  • #40821
    Living Miracles
    Participant

    Hello,

    Two days ago we upgraded the BPS Pro plugin from 15.8 to 15.9 across 30+ sites. Ever since, we’ve been seeing the following recurring PHP Warnings and Fatal Errors:

    2021-10-15 19:55:03 UTC [apache][php7:warn] [pid 54627] [client ##.###.###.###:#####] PHP Warning: file_put_contents(/home/customer/www/example.net/public_html/wp-content/plugins/bulletproof-security/admin/htaccess/plugins-htaccess-master.txt): failed to open stream: Too many open files in /home/customer/www/example.net/public_html/wp-content/plugins/bulletproof-security/includes/firewall-autopilot.php on line 1117
    ‮
    2021-10-15 19:55:03 UTC [apache][php7:warn] [pid 54627] [client ##.###.###.###:#####] PHP Warning: file_put_contents(/home/customer/www/example.net/public_html/wp-content/plugins/bulletproof-security/admin/htaccess/plugins.htaccess): failed to open stream: Too many open files in /home/customer/www/example.net/public_html/wp-content/plugins/bulletproof-security/includes/firewall-autopilot.php on line 1127
    ‮
    2021-10-15 19:55:03 UTC [apache][php7:warn] [pid 54627] [client ##.###.###.###:#####] PHP Warning: file_put_contents(/home/customer/www/example.net/public_html/wp-content/plugins/bulletproof-security/admin/htaccess/plugins-allow-from.txt): failed to open stream: Too many open files in /home/customer/www/example.net/public_html/wp-content/plugins/bulletproof-security/includes/firewall-autopilot.php on line 1138
    ‮
    2021-10-15 19:55:03 UTC [apache][php7:warn] [pid 54627] [client ##.###.###.###:#####] PHP Warning: file_get_contents(/home/customer/www/example.net/public_html/wp-content/plugins/bulletproof-security/admin/htaccess/plugins-allow-from.txt): failed to open stream: Too many open files in /home/customer/www/example.net/public_html/wp-content/plugins/bulletproof-security/includes/firewall-autopilot.php on line 1172
    ‮
    2021-10-15 19:55:03 UTC [apache][php7:warn] [pid 54627] [client ##.###.###.###:#####] PHP Warning: file_get_contents(/home/customer/www/example.net/public_html/wp-content/plugins/bulletproof-security/admin/htaccess/plugins.htaccess): failed to open stream: Too many open files in /home/customer/www/example.net/public_html/wp-content/plugins/bulletproof-security/includes/firewall-autopilot.php on line 1173
    ‮
    2021-10-15 19:55:03 UTC [apache][php7:warn] [pid 54627] [client ##.###.###.###:#####] PHP Warning: file_put_contents(/home/customer/www/example.net/public_html/wp-content/plugins/bulletproof-security/admin/htaccess/plugins.htaccess): failed to open stream: Too many open files in /home/customer/www/example.net/public_html/wp-content/plugins/bulletproof-security/includes/firewall-autopilot.php on line 1202
    ‮
    2021-10-15 19:55:03 UTC [apache][php7:error] [pid 54627] [client ##.###.###.###:#####] PHP Fatal error: Uncaught UnexpectedValueException: RecursiveDirectoryIterator::__construct(/home/customer/www/example.net/public_html/wp-content/plugins): failed to open dir: Too many open files in /home/customer/www/example.net/public_html/wp-content/plugins/bulletproof-security/includes/hidden-plugin-folders-cron.php:209\nStack trace:\n#0 /home/customer/www/example.net/public_html/wp-content/plugins/bulletproof-security/includes/hidden-plugin-folders-cron.php(209): RecursiveDirectoryIterator->__construct('/home/customer/...')\n#1 /home/customer/www/example.net/public_html/wp-content/plugins/bulletproof-security/includes/hidden-plugin-folders-cron.php(147): bpsPro_hidden_plugins_check_alert()\n#2 /home/customer/www/example.net/public_html/wp-includes/class-wp-hook.php(303): bpsPro_hidden_plugins_check()\n#3 /home/custom

    When these occur, the sites temporarily have a critical error on them and are inaccessible. The errors do not occur when we disable the PFW AutoPilot mode. Also, as a test, we restored one site using a backup from before we updated BPS Pro (i.e., when the plugin version was 15.8), and this also seemed to resolve the errors.

    Is this an issue with version 15.9 and, if so, do you have any recommendations with how to solve it?

    Alternatively, we could try rolling back the BPS Pro plugin from 15.9 to 15.8 on our remaining sites as a temporary workaround. However, for various reasons, we aren’t able to easily restore many of our other sites. In this case, can you tell us the simplest way to revert the BPS Pro version from 15.9 to 15.8 (without doing a full-site restore)?

    #40822
    AITpro Admin
    Keymaster

    Nothing has changed in BPS Pro 15.9 or 15.8 or 15.7, etc. that would cause these php errors.  So what that means is the root problem has to do with something else and you are seeing that problem occurring now.  Or in other words it is coincidental to whatever the root problem is.  Another factor is that this error is only occurring on your host server/websites.  No one else is reporting these php errors.

    The most common answer to this type of problem is this:  “This can be a limitation on the server where the code is running. Every operating system only allows for a certain number of open files/handles/sockets. This limit is usually further reduced when the server is virtualized.”.

    But that answer most likely does not apply in your case unless of course you have a lot of websites installed on a host server and you are reaching the maximum number of files allowed to be open at a time.  The number of files that BPS Pro is opening during any given processing is going to be very low relative to a typical default ulimit value > https://www.ibm.com/docs/en/watson-explorer/10.0.0?topic=tasks-setting-ulimit-values

    What I recommend is that you contact your web host support and ask them to check your server’s ulimit value.  If the ulimit value is set too low then have them increase it.  What is more likely happening is that this is a cumulative problem.  ie plugin x + plugin y + plugin z + theme a is exceeding your max ulimit.

    #40823
    Living Miracles
    Participant

    Thank you, this is very helpful information you were able to share with us! We actually discovered that the issue is not occurring on every single one of our sites. Since the single site restore test we did seemed to resolve the issue (no errors even after re-updating the BPS Pro plugin to 15.9), we’re thinking that the 15.8 to 15.9 update may have been incomplete/disrupted and somehow led to this issue. We would still like to try rolling back the plugin to 15.8 on the affected sites if possible. Is this possible without doing a full-site restore?

    #40824
    AITpro Admin
    Keymaster

    To eliminate the possibility of a bad/incomplete BPS Pro installation you can reinstall BPS Pro using the built-in BPS Pro Upload Zip installer under the Setup Menu > Upload Zip Install.  Download a new BPS Pro zip file from the AITpro secure download area.  I don’t recommend that you roll BPS Pro back to a previous version or try to do a site restore for this reason:  Nothing has changed in any BPS Pro versions for years that would start to cause these types of php errors.  So rolling back to any previous version of BPS Pro would not make any difference.  Most likely the root problem is that this is a cumulative problem. ie BPS Pro is attempting to open a few files, but something else you have installed is already pushing your server to the maximum allowed limit for open files.  What would be more logical to do would be to eliminate any new plugins you have installed recently.  Example problem scenario:  If you recently installed a plugin that is not using fclose() properly to close files then those files would be considered open.  If the problem is a looping problem then the problem would repeat exponentially.

    #40825
    AITpro Admin
    Keymaster

    What I recommend that you try first is to switch your PHP 7.x version to PHP 8.x.x in your web host control panel.  It could simply be that your current PHP server build/installation is fubar.  Also PHP 8.0.x is 18.47% faster than PHP 7.4.

    #40876
    Living Miracles
    Participant

    Thank you very much for the advice. We’ve been seeking support elsewhere as well but yours have been the only recommendations that really seem to appreciate the situation for what it is.

    Unfortunately, after some extensive troubleshooting (including all the usual standard/preliminary tests to rule out plugin conflicts, etc.), it’s still not clear what is/was happening on our sites. As you suggested, we contacted our host (SiteGround) and they did raise the ulimit to a much higher value (from 400 to 6000), though that didn’t seem to make any difference (i.e., the errors continued at the same rate). SiteGround was also quite clear that no server changes had been made on their end. We also tried updating some of our sites to PHP 8.x.x, but that didn’t seem to make any difference either (so, for the sake of troubleshooting, we switched the sites back to PHP 7.x, though we do plan to do a permanent upgrade to PHP 8.x.x at some point soon). Interestingly, several days later, the errors mysteriously started decreasing and eventually stopped completely.

    Then, a couple days after that, we noticed errors (thanks to ARQ) logged in WP core files for three of our sites:

    File: /home/customer/www/example.net/public_html/wp-admin/includes/php_errorlog -- Quarantine Time: 2021-10-23 19:27:02
    File: /home/customer/www/example.net/public_html/wp-includes/Requests/Transport/php_errorlog -- Quarantine Time: 2021-10-23 18:47:03
    File: /home/customer/www/example.net/public_html/wp-includes/SimplePie/Cache/php_errorlog -- Quarantine Time: 2021-10-23 18:17:02
    File: /home/customer/www/example.net/public_html/wp-includes/IXR/php_errorlog -- Quarantine Time: 2021-10-23 17:07:02

    According to SiteGround, these issues must be site-specific (as opposed to a server issue) since all sites are supposedly completely isolated from each other. So, we resumed checking all recent plugin/theme/etc. updates but still couldn’t find any commonalities in the affected sites.

    A day later, these errors stopped (without us having done anything).

    Then, three days after that, we started seeing new errors across about 3/4 of our sites:

    PHP Fatal error: require_once(): Failed opening required 'ABSPATHwp-admin/includes/class-wp-privacy-requests-table.php' (include_path='.:/usr/local/php74/pear') in /home/customer/www/example.net/public_html/wp-admin/includes/class-wp-privacy-data-export-requests-list-table.php on line 11
    PHP Fatal error: Uncaught Error: Class 'WP_Filesystem_Base' not found in /home/customer/www/example.net/public_html/wp-admin/includes/class-wp-filesystem-ftpsockets.php:16
    PHP Fatal error: Uncaught Error: Class 'WP_List_Table' not found in /home/customer/www/example.net/public_html/wp-admin/includes/class-wp-comments-list-table.php:18
    PHP Fatal error: Uncaught Error: Class 'WP_Privacy_Data_Export_Requests_List_Table' not found in /home/customer/www/example.net/public_html/wp-admin/includes/deprecated.php:1530

    These stopped occurring (again, without us doing anything) five days ago.

    So, we’re now in “observation mode” to see what else might happen, but don’t really have any idea what is causing the various errors (or whether they’re even related to each other or not) or how to proceed with troubleshooting.

    P.S. I don’t necessarily expect any additional support but I wanted to post this reply as it does feel like we’ve been able to rule out BPS Pro as a potential cause. Thanks again for your help.

    #40877
    AITpro Admin
    Keymaster

    Maybe your WordPress installation is damaged or incomplete.  Try reinstalling WP on the Dashboard > Updates page.  Intermittent problems are typically related to these things:  php memory/cache/caching plugins/CDN’s/VPN’s/Proxy’s/Load Balancers/Host server problems (new security measures added on Host server (ModSecurity, etc.), DNS server/DNS configuration problem, MySQL server timeout, server overloaded, etc.), /Browser problems (corrupt cache, Sessions, Cookies, add-on, extension)/ISP (connectivity)/CloudFlare, Incapsula, etc.

    This is an interesting clue: PHP Fatal error: require_once(): Failed opening required 'ABSPATHwp-admin/includes/class-wp-privacy-requests-table.php.... The path is displaying the ABSPATH constant instead of the actual path: /home/customer/www/example.net/public_html/. Taking a logical guess the problem may be related to server-side caching that did not correctly process the code.

    #40878
    Living Miracles
    Participant

    That’s a good catch, thanks for pointing that out!

    #40903
    AITpro Admin
    Keymaster

    I’m seeing the “failed to open stream: Too many open files” on another person’s site on SiteGround hosting.  Also the Setup Wizard crashes at the final AutoRestore file copy stage.  I tried both the new SG fast cloud server option and the standard PHP option and the same Setup Wizard crash occurred.  As you can see from the php errors the problem is not isolated to any particular plugin.  So at this point this appears to be a server problem.  Host servers typically have built-in overload features that automatically shutdown the server if the server thinks that too many resources are being used.  I’m not one to pass blame, but this sure looks like an SG server-side issue.  😉

    [10-Nov-2021 16:17:48 UTC] PHP Warning: file_put_contents(/home/customer/www/walkingsports.com/public_html/wp-content/plugins/bulletproof-security/admin/htaccess/plugins.htaccess): failed to open stream: Too many open files in /home/customer/www/walkingsports.com/public_html/wp-content/plugins/bulletproof-security/includes/firewall-autopilot.php on line 1202
    [10-Nov-2021 16:19:09 UTC] PHP Warning: require_once(/home/customer/www/walkingsports.com/public_html/wp-content/plugins/complianz-terms-conditions/config/documents/documents.php): failed to open stream: Too many open files in /home/customer/www/walkingsports.com/public_html/wp-content/plugins/complianz-terms-conditions/config/class-config.php on line 40
    [10-Nov-2021 16:19:09 UTC] PHP Fatal error: require_once(): Failed opening required '/home/customer/www/walkingsports.com/public_html/wp-content/plugins/complianz-terms-conditions//config/documents/documents.php' (include_path='.:/usr/local/php74/pear') in /home/customer/www/walkingsports.com/public_html/wp-content/plugins/complianz-terms-conditions/config/class-config.php on line 40
    [10-Nov-2021 16:19:09 UTC] PHP Fatal error: require_once(): Failed opening required '/home/customer/www/walkingsports.com/public_html/wp-content/plugins/complianz-gdpr-premium//pro/config/documents/AU/processing-agreement.php' (include_path='.:/usr/local/php74/pear') in /home/customer/www/walkingsports.com/public_html/wp-content/plugins/complianz-gdpr-premium/pro/config/includes.php on line 37
    [10-Nov-2021 16:20:08 UTC] PHP Warning: require_once(/home/customer/www/walkingsports.com/public_html/wp-content/plugins/geodirectory/includes/widgets/class-geodir-widget-ninja-forms.php): failed to open stream: No such file or directory in /home/customer/www/walkingsports.com/public_html/wp-content/plugins/geodirectory/includes/class-geodir-compatibility.php on line 1323
    [10-Nov-2021 16:20:08 UTC] PHP Fatal error: require_once(): Failed opening required '/home/customer/www/walkingsports.com/public_html/wp-content/plugins/geodirectory/includes/widgets/class-geodir-widget-ninja-forms.php' (include_path='.:/usr/local/php74/pear') in /home/customer/www/walkingsports.com/public_html/wp-content/plugins/geodirectory/includes/class-geodir-compatibility.php on line 1323
    [10-Nov-2021 16:20:14 UTC] PHP Warning: require_once(/home/customer/www/walkingsports.com/public_html/wp-content/plugins/complianz-terms-conditions/class-document.php): failed to open stream: Too many open files in /home/customer/www/walkingsports.com/public_html/wp-content/plugins/complianz-terms-conditions/complianz-terms-conditions.php on line 103
    [10-Nov-2021 16:20:14 UTC] PHP Fatal error: require_once(): Failed opening required '/home/customer/www/walkingsports.com/public_html/wp-content/plugins/complianz-terms-conditions/class-document.php' (include_path='.:/usr/local/php74/pear') in /home/customer/www/walkingsports.com/public_html/wp-content/plugins/complianz-terms-conditions/complianz-terms-conditions.php on line 103
    [10-Nov-2021 16:34:11 UTC] PHP Warning: require_once(/home/customer/www/walkingsports.com/public_html/wp-content/plugins/popup-maker/classes/Utils/Logging.php): failed to open stream: Too many open files in /home/customer/www/walkingsports.com/public_html/wp-content/plugins/popup-maker/popup-maker.php on line 70
    [10-Nov-2021 16:34:11 UTC] PHP Fatal error: require_once(): Failed opening required '/home/customer/www/walkingsports.com/public_html/wp-content/plugins/popup-maker/classes/Utils/Logging.php' (include_path='.:/usr/local/php74/pear') in /home/customer/www/walkingsports.com/public_html/wp-content/plugins/popup-maker/popup-maker.php on line 70
    [10-Nov-2021 16:34:12 UTC] PHP Warning: require_once(/home/customer/www/walkingsports.com/public_html/wp-content/plugins/complianz-terms-conditions/config/countries.php): failed to open stream: Too many open files in /home/customer/www/walkingsports.com/public_html/wp-content/plugins/complianz-terms-conditions/config/class-config.php on line 37
    [10-Nov-2021 16:34:12 UTC] PHP Fatal error: require_once(): Failed opening required '/home/customer/www/walkingsports.com/public_html/wp-content/plugins/complianz-terms-conditions//config/countries.php' (include_path='.:/usr/local/php74/pear') in /home/customer/www/walkingsports.com/public_html/wp-content/plugins/complianz-terms-conditions/config/class-config.php on line 37
    #40904
    Living Miracles
    Participant

    Wow, very interesting. Thanks for keeping us in mind and passing along the info!

    #40912
    Ian
    Participant

    Hi,

    We have had a bit of a ping pong match trying to get Siteground to seriously look at this issue.

    Finally today it was escalated to their advanced Technical Support team.

    This is their findings and we would really appreciate your input…

    First, I would like to clarify that my colleague Nadya was referring to the WordPress application and its installed plugins. In some cases, the application is trying to open too many files, such as extracting a zip file. For example, this piece of PHP code could potentially generate the error “too many open files” as the extracted files are actually open sockets. When their number reached 4000, the PHP script will be terminated:

    for ($index = 0; $index < $zip->numFiles; $index++)
    
    {
    
    $file = $zip->getNameIndex($index);
    
    if (substr($file, -1) === '/')
    
    {
    
    continue;
    
    }

    One possible fix is to add a delay when listing an extracting the files:

    for ($index = 0; $index < $zip->numFiles; $index++)
    
    {
    
    $file = $zip->getNameIndex($index);
    
    usleep(1000);
    
    if (substr($file, -1) === '/')
    
    {
    
    continue;
    
    }

    The usleep will add 1000 microseconds between the extraction and listing of the files, which will prolong the execution of the script and extraction of the files, but will avoid the error for too many open files. Here is another example with a similar code but with a different usleep time:

    for( $i = 0; $i < $zip->numFiles; $i++ ){
    
    $stat = $zip->statIndex( $i );
    
    echo "Extract " . $stat['name'] . "\n";
    
    usleep(2000);
    
    if ($zip->extractTo(__DIR__ , $stat['name']) == false) {
    
    echo "Exctract ".$stat['name']." fail\n";
    
    }
    
    }
    

    Please note that both examples are taken from developers having an error for too many open files on our servers. The discussion on https://forum.ait-pro.com/forums/topic/php-errors-in-bps-pro-15-9/ seems to indicate that the error occurs when AutoRestore file copy stage is performed. If this involves extracting an archive, the issue could possibly be fixed by adding a delay as in the provided examples.

    This is why it is important for a professional developer who can check the application and optimize the source code of the installed plugins and themes to make sure that the open sockets do not reach the server limit. Our system administrators have already increased the maximum allowed open files from 4000 to 6500. We would not be able to increase the limit further as this could reduce the performance and cause server crash during high load events.

    #40913
    AITpro Admin
    Keymaster

    @ Ian – I don’t believe the problem is actually what it appears to be.  I don’t think the real problem is that too many files are being opened.  The reasons for that thinking is that BPS Pro is installed on 100’s of web hosts worldwide and this specific problem is only occurring on SiteGround servers.  Also this is a new issue that only started occurring in the last couple of months on SiteGround.  I have many users that have been using SiteGround for years without this issue occurring.  I am by no means trying to blame SiteGround in any way.  I will help in any way that I can to provide information from my side of things, which is:  This is a relatively new issue that is only occurring on SiteGround servers in the last couple of months.  I believe what is actually happening is some mechanism on the server that determines if too many files are open is malfunctioning.  This has got to be some technical server configuration thing.

    With that said, it could be possible that another plugin that is installed is actually really opening too many files, but I still believe this is a server configuration malfunction issue.  BPS Pro would not be doing that.  The AutoRestore phase of the Setup Wizard copies Live website files to the AutoRestore backup folder.  There is no zipping/archiving involved, only the php copy() function using php recursive iterator classes.  So since this issue is only occurring on SiteGround servers then once again it would be very unlikely that BPS Pro is the cause of this issue.

    The most logical way to troubleshoot this issue would be to look at all the server configuration mechanisms that are involved in monitoring and checking how many files are open.  The php error message – “Failed to open stream: Too many open files” is only a symptom of the issue (is probably not correct/accurate) and not the actual problem.

    #40916
    AITpro Admin
    Keymaster

    @ Ian – Example: The actual problem could be “…the process is leaking file descriptors…”.

    https://www.ibm.com/support/pages/how-diagnose-too-many-open-files-issues

    Applications or servers can sometimes fail with an error indicating that
    there are too many open files for the current process. Most of the time
    the problem is due to a configuration too small for the current needs.
    Sometimes as well it might be that the process is ‘leaking’ file descriptors.
    In other words, the process is opening files but does not close them leading
    to exhaustion of the available file descriptors.

    If you face the ‘too many open files’ error here are a few things you can
    try to identify the source of the problem.

    – 1 – Check the current limits.
    – 2 – Check the limits of a running process.
    – 3 – Tracking a possible file descriptors leak.
    – 4 – Tracking open files in real time.
    – 5 – Specific to DB2.
    – 6 – Extra notes.

    #40917
    AITpro Admin
    Keymaster

    @ Ian – After doing a little more research on this issue – “leaking” file descriptors is my best guess.  Basically what is most likely happening on the SiteGround server or servers is this – all available file handles for the current process have been used (this includes sockets as well).  If files are not seen as being closed due to a leaking file descriptor problem then even opening 1 additional file will trigger the too many open files error, which is exactly the problem that is occurring.  So pass this information on to SiteGround support and have them look into this first.  I feel very confident that this is most likely the root problem.

    #40933
    Ian
    Participant

    Hi,

    Just wanted to say thanks for all of the input.

    We have been monitoring this for the past few days and have not had the error.

    So fingers crossed.

Viewing 15 posts - 1 through 15 (of 15 total)
  • You must be logged in to reply to this topic.