View Issue Details

IDProjectCategoryView StatusLast Update
0000355Apache 2.x Bugpublic2020-08-24 03:14
Reporterltyndale Assigned Topsmedley  
PrioritynormalSeveritycrashReproducibilityrandom
Status closedResolutionduplicate 
Product Versionv2.2.1 
Summary0000355: Random crashes with Apache 2.2.11
DescriptionVersion: 2.2.11 (February 23 version)

Apache will randomly crash. The server (httpd.exe) will shut down entirely. In the error log is the message:

[Wed Jun 17 22:51:36 2009] [error] (OS 10053)Software caused connection abort: apr_socket_accept
[Wed Jun 17 22:51:36 2009] [error] (OS 10053)Software caused connection abort: apr_socket_accept
[Wed Jun 17 22:51:37 2009] [notice] caught SIGTERM, shutting down

I notice that on the project page at http://www.smedley.info/os2ports/index.php?page=apache2 it suggests that the apr_socket_accept issue should be resolved with this build, but I'm unfortunately still seeing it.

TagsNo tags attached.

Relationships

duplicate of 0000425 closedpsmedley Unknown crashes with Apache 2.2.11 

Activities

ltyndale

2009-06-20 22:23

reporter   ~0001352

Well, it seems this issue also exists in the 2.2.10 build of Apache.

I find that after about 14 hours of running using the PHP module and serving Wordpress, Apache gets excessively slow, then locks up, after eating up whatever free memory is available and opening up lots of sockets, regardless of what settings I use within httpd.conf or the TCP/IP keepalive settings. Also when it locks, if I try and kill it, it won't kill. If I try and reboot with either a command line reboot (setboot /b) or a CTRL-ALT-DEL, the machine will start the reboot process and then freeze. The only way to reboot seems to be to kill the power.

Interestingly it'll still serve static files, but anything PHP it'll just choke on. I don't know if this is a PHP issue or an Apache issue.

I've tried different builds of PHP - 5.2.9, 5.2.10 (the one just posted), 5.2.6, 5.2.8. Each time I've switched things around I've made sure to replace the modphp5.dll in the apache modules folder, to ensure everything is loaded with the proper versions.

I'm not sure if this is a PHP issue or if it is related to how the Apache 2.2.x codebase interfaces with PHP (the modphp5.dll), or what. When I run PHP as a CGI using Dink's Web/2, I have not run into this difficulty. The main reason why I've switched to Apache is to get the rewrite URL's which Apache can do and Web/2 can't do.

I should mention I've also tried this on 2 different computers (thinking that one of them might have some rogue DLL, bad memory, hard disk issue, or something else that I wasn't seeing). I ran into the same problem on both computers.

Any ideas? I may have to try one of the really old versions of Apache and see if that does it too or not.

ltyndale

2009-06-30 23:57

reporter   ~0001359

Last edited: 2009-07-01 00:03

After trying multiple versions of Apache, it looks like this could be an issue with a priority inversion within Apache - PHP.

After a bunch of troubleshooting and trying things out on 3 different machines (each with similar difficulties), it occurred to me that on each machine I have PRIORITY=ABSOLUTE set in the config.sys. Normally for servers I've found that setting priority to absolute, and then setting the individual priority of each app can increase server performance by 25 - 40%. But if there ever is a priority inversion in an app (if an app tries to set its own priorities for threads, and depends on those priorities as it waits for tasks to finish) it can lead to applications that get hung up. So on one of the machines I set up to try and troubleshoot this issue with Apache / PHP randomly freezing up, I tried setting PRIORITY=DYNAMIC in config.sys.

Apache 2.2.8 with PHP 5.2.4 has now run for over 24 hours without any issues, without locking up, and it can even be killed with a CTRL-C. The machine reboots properly with a CTRL-ALT-DEL or a Setboot /b when Apache is running (which it previously would not do).

I'm going to next try the latest builds of Apache / PHP and see if they work as they should with the priority=dynamic setting.

psmedley

2009-07-30 21:48

administrator   ~0001384

I'm curious if http://smedley.info/httpd-2.2.11-os2-20090728.zip helps in any way

ssha

2009-08-08 15:47

reporter   ~0001389

So far no apr_socket_accept messages, but HTTPD (and PHP) still bomb out after a bit of work. There seems to be a problem with memory usage.

Assuming that all of Paul's builds are being used (I am on MySQL 5.0 still for the moment):

1. MySQL seems to get most of its memory from the shared arena (between 50-60Mb)
2. HTTPD here becomes exhausted - with logging enabled, PHP reports trying to allocate memory, but the request is refused (332922880 allocated, tried to allocate 500001 bytes)
3. On the system here, HTTPD was consuming 3316, 345424 and 345400 kbytes of memory usage (private) and a 1 or 2 Mb in total of shared, when the above PHP error is logged - HTTPD seems to be running three times
4. At this point, Thesus shows the gap between the private and shared arenas is 0Mb, and the total amount of free shared memory is 6.1M in total.
5. Free private memory was still 511Mb.

Questions:

1. Why does MySQL (at least 5.0) seem to get most of its memory from the shared arena?
2. Why does HTTPD seem to be relying on shared memory, when Thesus shows it using mostly private? Is this how LIBC works?
3. Is there a memory free problem in HTTPD?

Hope this helps, remote VNC available to my system if needed.

Thanks,
Stephen

ltyndale

2009-08-08 21:21

reporter   ~0001391

I too have not run into the apr_socket message with this build, but have run into the random lockups/freezing. This only seems to occur with PHP files that are accessing a MySQL database, and usually with PHP files that are more demanding on system memory. I particularly see this with Wordpress when logged both into the backend admin area and have the front-end website up for testing/viewing changes made to the content.

I should note that some PHP based stuff (such as Limesurvey, which is fairly light on memory usage within its scripts) seems to work fine and can run for months without causing any trouble. And it doesn't occur with static HTML files - in fact, when its frozen up for PHP based stuff, static HTML files still get served fine.

I'm not an expert on this, but based on Stephen's information it does sound like a memory issue, somewhere memory is getting used but not released. Based on Stephen's post and my own experience of static HTML files getting served but PHP files being non-responsive, it does sound like a memory issue, that PHP just isn't getting the memory it needs to process the file(s).

ssha

2009-08-08 21:26

reporter   ~0001392

I should also add that my max memory under PHP.INI is set to 768M (and whilst there is still plenty of private memory available) the unavailability of shared memory seems to hit PHP.

I'd also be interested in how/why LIBC allocates/shares memory... why do both HTTPD processes running consume a lot of memory, and can LIBC not be forced to always used private memory?

ssha

2009-08-08 21:47

reporter   ~0001393

As daft as it sounds, I am not sure if it really is a 'leak' as such - at least, not with the app I am using (Jinzora) - it really does want to use that much memory (i.e. re-continuing a failed track search instantly re-consumes the 300Mb of memory). The problem seems to be that this memory seems to need to come out of the shared arena (by LIBC), even tho HTTPD's memory usage is in the *high* arena. So does LIBC also have a copy of the assigned data in memory, in the low arena?

ssha

2009-08-09 12:46

reporter   ~0001395

OK, not sure that HTTPD's memory usage is in the *high* arena - just coz Thesus shows its data as private, doesn't mean it is high.

Starting to do any worth with a fresh Jinzora install (sorry to have to relate this to Jinzora, but its my only test case at the moment) as soon as I start to do a lyric update, for example, all of OS/2's shared memory dissappears (SHMEMMON shows 11mb available) and the two child HTTPD processes grow huge.

Can't tell tho which DLL is using all the memory (i.e. LIBC063, PHP or HTTPD) - is there an easy way to see this?

I have also seen, just once, Apache hang - in the middle of doing not very much (running PHPGET info?) - no sys3175, no error message. CTRL-C caused sigterm, and after a restart everything is OK.

Paul - do you want me to open up a seperate issue for the memory usage? There seems to be a 'runaway' problem with memory usage - wonder if lots of these problems could be due to an exhaustion of the shared memory?

Don't really know enough about what I'm doing with Theseus to find out more.

Remote access to my test system available via VNC, if that helps...

ssha

2009-08-09 22:44

reporter   ~0001398

More testing...

The high memory usage is caused by PHP, not HTTPD. I guess Theseus previously showed the high memory usage attributed to HTTPD only because httpd.exe loads php5.dll.

Running the same PHP script from the command-line (admittedly, a slightly old version of PHP as the latest has no CLI build) causes PHP to use almost all available lower memory.

It seems that returned datasets by PHP from MySQL are being held in the shared arena. Can PHP's memory allocation calls not be modified to use the 'high' flag? (I seem to remember reading this somewhere).

At present, all PHP datasets must be below about 300Mb, otherwise the shared memory becomes exhausted (causing all kinds of weird HTTPD problems).

ssha

2009-08-11 12:43

reporter   ~0001402

Calls to php's get memory usage just before a crash after a test script has been running for a while show usage of 19Mb (and slowly increaing), then the out of memory is immediately reported with 44Mb used, and exhausted. Sounds like a library problem?

Not sure what else I can do to test, of course, the PHP memory problems are only a theory wrt to the original Apache apr_socket problem - but if PHP really does consume the entire shared arena, then all kinds of problems might happen :)

LewisR

2009-10-06 06:20

developer   ~0001416

Following onto my various discussions on the Apache/2 list, Lorne, how many Listen directives do you have in your various conf files?

Current uptime on my server after removing the second (or two) generic Listen directives (Listen 81) as resulted in uptime now of 21+ hours, with *no* daemon restarts and *no* errors written to the log (other than the debug information).

psmedley

2010-07-11 03:13

administrator   ~0001639

Please try http://download.smedley.info/httpd-2.2.15-os2-20100711.zip

psmedley

2010-07-11 03:13

administrator   ~0001640

Please try http://download.smedley.info/httpd-2.2.15-os2-20100711.zip

psmedley

2011-12-31 08:26

administrator   ~0002105

Duplicate of 425

Issue History

Date Modified Username Field Change
2009-06-18 05:26 ltyndale New Issue
2009-06-20 22:23 ltyndale Note Added: 0001352
2009-06-30 23:57 ltyndale Note Added: 0001359
2009-07-01 00:01 ltyndale Note Edited: 0001359
2009-07-01 00:03 ltyndale Note Edited: 0001359
2009-07-30 21:48 psmedley Note Added: 0001384
2009-08-08 15:47 ssha Note Added: 0001389
2009-08-08 21:21 ltyndale Note Added: 0001391
2009-08-08 21:26 ssha Note Added: 0001392
2009-08-08 21:47 ssha Note Added: 0001393
2009-08-09 12:46 ssha Note Added: 0001395
2009-08-09 22:44 ssha Note Added: 0001398
2009-08-11 12:43 ssha Note Added: 0001402
2009-10-06 06:20 LewisR Note Added: 0001416
2010-07-11 03:13 psmedley Note Added: 0001639
2010-07-11 03:13 psmedley Note Added: 0001640
2010-07-11 03:13 psmedley Assigned To => psmedley
2010-07-11 03:13 psmedley Status new => feedback
2011-12-31 08:25 psmedley Relationship added duplicate of 0000425
2011-12-31 08:26 psmedley Note Added: 0002105
2011-12-31 08:26 psmedley Status feedback => resolved
2011-12-31 08:26 psmedley Resolution open => duplicate
2020-08-24 03:14 psmedley Status resolved => closed