View Issue Details

IDProjectCategoryView StatusLast Update
0000736Apache 2.x Bugpublic2021-07-26 06:19
ReporteremaxAssigned ToSteven Levine 
PriorityurgentSeverityblockReproducibilitysometimes
Status assignedResolutionopen 
PlatformVM guest on Vbox 2 cores, 3GBramOSAOSOS Version503
Product Version2.4.x 
Target VersionFixed in Version 
Summary0000736: Apache 2.4.46 build (1th/4/2021) with php 5.6.40 sometimes don't restart
DescriptionServer Version: Apache/2.4.46 (OS/2) OpenSSL/1.1.1j PHP/5.6.40
Server Built: Apr 1 2021 14:58:19

Hi all,

sometimes apache don't start/restart with this error:

02816271 01 ff 0000 Asrt: Assertion Failed!!!
02816271 01 ff 0000 Asrt: Function: <NULL>
02816271 01 ff 0000 Asrt: File: D:/Users/dmik/rpmbuild/BUILD/libcx-0.6.6/src/shared.c
02816271 01 ff 0000 Asrt: Line: 189
02816271 01 ff 0000 Asrt: Expr: arc == NO_ERROR
02816271 01 ff 0000 Asrt: 105

i've to setboot /b the VM to have it back starting again

this is a major issue :(

sorry
TagsNo tags attached.

Activities

emax

2021-06-04 19:32

reporter   ~0003706

this kind of crash/dump don't produce any exceptQ dump file


apache error log:

[Fri Jun 04 10:28:17.649000 2021] [mpm_mpmt_os2:notice] [pid 493:tid 1] (OS 10035)Resource temporarily unavailable: apr_socket_accept
[Fri Jun 04 10:30:47.086000 2021] [mpm_mpmt_os2:notice] [pid 493:tid 1] (OS 10035)Resource temporarily unavailable: apr_socket_accept
Assertion info: 105
Assertion failed: arc == NO_ERROR, file D:/Users/dmik/rpmbuild/BUILD/libcx-0.6.6/src/shared.c, line 564

Killed by SIGABRT
pid=0x01f2 ppid=0x01b4 tid=0x0001 slot=0x007d Assertion info: 105
Assertion failed: arc == NO_ERROR, file D:/Users/dmik/rpmbuild/BUILD/libcx-0.6.6/src/shared.c, line 564

Killed by SIGABRT
pid=0x01f4 ppid=0x01b4 tid=0x0012 slot=0x00b9 pri=0x0200 mc=0x0001 ps=0x0010
D:\APACHE\BIN\HTTPD.EXE
Process dumping was disabled, use DUMPPROC / PROCDUMP to enable it.
[Fri Jun 04 10:52:07.031000 2021] [mpm_mpmt_os2:error] [pid 502:tid 1] (OS 105)The previous ownership of this semaphore has ended. : Sem owner died pidSemaphoreOwner = 501, tidSemaphoreOwner = 1, ulRequestCount = 1 rc = 105 at ap_mpm_child_main #0
[Fri Jun 04 10:52:07.037000 2021] [mpm_mpmt_os2:error] [pid 502:tid 1] (OS 105)The previous ownership of this semaphore has ended. : AH00194: apr_socket_accept
[Fri Jun 04 10:52:07.028000 2021] [mpm_mpmt_os2:error] [pid 503:tid 1] (OS 105)The previous ownership of this semaphore has ended. : Sem owner died pidSemaphoreOwner = 501, tidSemaphoreOwner = 1, ulRequestCount = 1 rc = 105 at ap_mpm_child_main #0
[Fri Jun 04 10:52:07.038000 2021] [mpm_mpmt_os2:error] [pid 503:tid 1] (OS 105)The previous ownership of this semaphore has ended. : AH00194: apr_socket_accept

after reboot:

[Fri Jun 04 11:33:38.336000 2021] [mpm_mpmt_os2:error] [pid 67:tid 1] AH00200: DosGetNamedSharedMem returned 0
[Fri Jun 04 11:33:38.365000 2021] [mpm_mpmt_os2:notice] [pid 67:tid 1] AH00206: Apache/2.4.46 (OS/2) OpenSSL/1.1.1j PHP/5.6.40 configured -- resuming normal operations

psmedley

2021-06-04 19:33

administrator   ~0003707

This is a libcx issue and needs to be reported to bww

psmedley

2021-06-04 20:24

administrator   ~0003708

Suggest reporting at https://github.com/bitwiseworks/libcx/issues

psmedley

2021-06-04 20:25

administrator   ~0003709

Similar to https://github.com/bitwiseworks/libcx/issues/82

emax

2021-06-04 20:29

reporter   ~0003710

done, thanks

https://github.com/bitwiseworks/libcx/issues/92

emax

2021-06-04 20:40

reporter   ~0003711

it seems i've an old libcx, of course "yum update libcx" don't work properly
it detect the old version, but faile on update

Download dei pacchetti:
Running Transaction Check
Test di transazione in corso
Errore nel controllo transazione:
  il file /@unixroot/usr/bin/pwd_mkdb.exe dell'installazione di libc-1:0.1.7-1.oc00.

emax

2021-06-04 20:50

reporter   ~0003712

i've updated it manually from version 0.6.6. to 0.7.0
let's see if it improves the situation

psmedley

2021-06-05 07:25

administrator   ~0003713

I normally just run 'yum update'

emax

2021-06-07 21:18

reporter   ~0003717

upgrading libcx0 from version 0.6.6. to 0.7.0 this issue seems to be fixed
please give me some more day to confirm

thanks

emax

2021-06-14 18:01

reporter   ~0003733

this crash after a number of urls "not found or unable to stat" gives a very interesting dump
but NO dumps in popuplog.os2, NO exceptQ dumps


[Sun Jun 13 10:49:10.372000 2021] [:error] [pid 6055:tid 22] [client X.Y.Z.A:31621] script '/apache/htdocs/mywebsite/wp-login.php' not found or unable to stat

Killed by SIGSEGV
pid=0x17a7 ppid=0x10f1 tid=0x0001 slot=0x00f3 pri=0x0200 mc=0x0001 ps=0x0010
D:\APACHE\BIN\HTTPD.EXE
PHP5 0:0078a90a
cs:eip=005b:1dbca90a ss:esp=0053:0022fbc0 ebp=0022fc08
 ds=0053 es=
Killed by SIGSEGV
pid=0x17bd ppid=0x10f1 tid=0x0001 slot=0x007f pri=0x0200 mc=0x0001 ps=0x0010
D:\APACHE\BIN\HTTPD.EXE
PHP5 0:00706dff
cs:eip=005b:1db46dff ss:esp=0053:0022fb80 ebp=0022fbc8
 ds=0053 es=0053 fs=150b gs=0000 efl=00010202
eax=00000000 ebx=24eefcc0 ecx=00000022 edx=00000000 edi=28684220 esi=200e4890
Process dumping was disabled, use DUMPPROC / PROCDUMP to enable it.
HTTPD.EXE
PHP5 0:0078a90a
cs:eip=005b:1dbca90a ss:esp=0053:0022fbc0 ebp=0022fc08
 ds=0053 es=0053 fs=150b gs=0000 efl=00010206
eax=00000000 ebx=0003fff0 ecx=00000000 edx=22738300 edi=22738308 esi=20111600
Process dumping was disabled, use DUMPPROC / PROCDUMP to enable it.
0022fbc8
 ds=0053 es=Assertion info: 105

===== LIBCx resource usage =====
Reserved memory size: 2097152 bytes
Committed memory size: 65536 bytes
Heap size total: 65120 bytes
Heap size used now: 7910 bytes
ProcDesc structs used now: 8
FileDesc structs used now: 17
SharedFileDesc structs used now: 10
===== LIBCx stats end =====
Assertion failed: arc == NO_ERROR, file D:/Users/dmik/rpmbuild/BUILD/libcx-0.7.0/src/shared.c, line 564

Killed by SIGABRT
pid=0x17ba ppid=0x10f1 tid=0x0001 slot=0x009b pri=0x0200 mc=0x0001 ps=0x0010
D:\APACHE\BIN\HTTPD.EXE
Process dumping was disabled, use DUMPPROC / PROCDUMP to enable it.
[Sun Jun 13 11:01:21.532000 2021] [mpm_mpmt_os2:error] [pid 6334:tid 1] (OS 10038)Socket operation on non-socket: AH00194: apr_socket_accept
Assertion info: 105

===== LIBCx resource usage =====
Reserved memory size: 2097152 bytes
Committed memory size: 65536 bytes
Heap size tAssertion info: 105

===== LIBCx resource usage =====
Reserved memory size: 2097152 bytes
Committed memory size: 65536 bytes
Heap size total: 65120 bytes
Heap size used now: 3492 bytes
ProcDesc structs used now: 5
FileDesc structs used now: 4
SharedFileDesc structs used now: 1
===== LIBCx stats end =====
Assertion failed: arc == NO_ERROR, file D:/Users/dmik/rpmbuild/BUILD/libcx-0.7.0/src/shared.c, line 564

Killed by SIGABRT
pid=0x18bc ppid=0x0010 tid=0x0003 slot=0x00b2 pri=0x0200 mc=0x0001 ps=0x0010
D:\APACHE\BIN\HTTPD.EXE
Process dumping was disabled, use DUMPPROC / PROCDUMP to enable it.
Assertion info: 105

===== LIBCx resource usage =====
Reserved memory size: 2097152 bytes
Committed memory size: 65536 bytes
Heap size tAssertion info: 105

===== LIBCx resource usage =====
Reserved memory size: 2097152 bytes
Committed memory size: 65536 bytes
Heap size total: 65120 bytes
Heap size used now: 2856 bytes
ProcDesc structs used now: 4
FileDesc structs used now: 3
SharedFileDesc structs used now: 1
===== LIBCx stats end =====
Assertion failed: arc == NO_ERROR, file D:/Users/dmik/rpmbuild/BUILD/libcx-0.7.0/src/shared.c, line 564

Killed by SIGABRT
pid=0x18bd ppid=0x0010 tid=0x0004 slot=0x0075 pri=0x0200 mc=0x0001 ps=0x0010
D:\APACHE\BIN\HTTPD.EXE
Process dumping was disabled, use DUMPPROC / PROCDUMP to enable it.
[Sun Jun 13 11:05:52.902000 2021] [mpm_mpmt_os2:error] [pid 67:tid 1] AH00200: DosGetNamedSharedMem returned 0
[Sun Jun 13 11:05:52.927000 2021] [mpm_mpmt_os2:notice] [pid 67:tid 1] AH00206: Apache/2.4.46 (OS/2) OpenSSL/1.1.1j PHP/5.6.40 configured -- resuming normal operations

Killed by SIGSEGV
pid=0x0044 ppid=0x0043 tid=0x0001 slot=0x005b pri=0x0200 mc=0x0001 ps=0x0010
D:\APACHE\BIN\HTTPD.EXE
PHP5 0:00706dff
cs:eip=005b:1db46dff ss:esp=0053:0022fb80 ebp=0022fbc8
 ds=0053 es=0053 fs=150b gs=0000 efl=00010202
eax=00000000 ebx=218f0d40 ecx=00000022 edx=00000000 edi=218f0d00 esi=200e4890
Process dumping was disabled, use DUMPPROC / PROCDUMP to enable it.
[client X.Y.Z.A:51006] script '/apache//mywebsite/wp-login.php' not found or unable to stat, referer: http://www.abcdef.com

Killed by SIGSEGV
pid=0x0048 ppid=0x0043 tid=0x0001 slot=0x0060 pri=0x0200 mc=0x0001 ps=0x0010
D:\APACHE\BIN\HTTPD.EXE
PHP5 0:00706dff
cs:eip=005b:1db46dff ss:esp=0053:0022fb80 ebp=0022fbc8
 ds=0053 es=0053 fs=150b gs=0000 efl=00010202
eax=00000000 ebx=219fa0c0 ecx=00000022 edx=00000000 edi=219f86a0 esi=200e4890
Process dumping was disabled, use DUMPPROC / PROCDUMP to enable it.

emax

2021-06-14 18:04

reporter   ~0003735

also updated ticket on github

https://github.com/bitwiseworks/libcx/issues/92

emax

2021-06-21 23:38

reporter   ~0003750

today i had another serious issue:

when this issue occurs, apache do not restart anymore and i've to seboot /b the server :-(

[Mon Jun 21 08:21:06.933000 2021] [mpm_mpmt_os2:notice] [pid 426:tid 1] (OS 10035)Resource temporarily unavailable: apr_socket_accept
[Mon Jun 21 08:22:01.642000 2021] [mpm_mpmt_os2:notice] [pid 428:tid 1] (OS 10035)Resource temporarily unavailable: apr_socket_accept
[Mon Jun 21 08:23:25.590000 2021] [mpm_mpmt_os2:notice] [pid 427:tid 1] (OS 10035)Resource temporarily unavailable: apr_socket_accept
[Mon Jun 21 08:26:35.729000 2021] [mpm_mpmt_os2:notice] [pid 424:tid 1] (OS 10035)Resource temporarily unavailable: apr_socket_accept
Assertion info: 105

===== LIBCx resource usage =====
Reserved memory size: 2097152 bytes
Committed memory size: 65536 bytes
Heap size total: 65120 bytes
Heap size used now: 7262 bytes
ProcDesc structs used now: 8
FileDesc structs used now: 14
SharedFileDesc structs used now: 8
===== LIBCx stats end =====
Assertion failed: arc == NO_ERROR, file D:/Users/dmik/rpmbuild/BUILD/libcx-0.7.0/src/shared.c, line 564

Killed by SIGABRT
pid=0x01b1 ppid=0x019a tid=0x0001 slot=0x0083 pri=0x0200 mc=0x0001 ps=0x0010
X:\APACHE\BIN\HTTPD.EXE
Process dumping was disabled, use DUMPPROC / PROCDUMP to enable it.
600
Process dumping was disabled, use DUMPPROC / PROCDUMP to enable it.
Assertion info: 105

===== LIBCx resource usage =====
Reserved memory size: 2097152 bytes
Committed memory size: 65536 bytes
Heap size total: 65120 bytes
Heap size used now: 4128 bytes
ProcDesc structs used now: 6
FileDesc structs used now: 5
SharedFileDesc structs used now: 1
===== LIBCx stats end =====
Assertion failed: arc == NO_ERROR, file D:/Users/dmik/rpmbuild/BUILD/libcx-0.7.0/src/shared.c, line 564

Killed by SIGABRT
pid=0x01b4 ppid=0x019a tid=0x0004 slot=0x00ea pri=0x0200 mc=0x0001 ps=0x0010
X:\APACHE\BIN\HTTPD.EXE
Process dumping was disabled, use DUMPPROC / PROCDUMP to enable it.
[Mon Jun 21 08:46:09.710000 2021] [mpm_mpmt_os2:error] [pid 438:tid 1] (OS 10038)Socket operation on non-socket: AH00194: apr_socket_accept
Assertion info: 105

===== LIBCx resource usage =====
Reserved memory size: 2097152 bytes
Committed memory size: 65536 bytes
Heap size Assertion info: 105

===== LIBCx resource usage =====
Reserved memory size: 2097152 bytes
Committed memory size: 65536 bytes
Heap size Assertion info: 105

===== LIBCx resource usage =====
Reserved memory size: 2097152 bytes
Committed memory size: 65536 bytes
Heap size total: 65120 bytes
Heap size used now: 2856 bytes
ProcDesc structs used now: 4
FileDesc structs used now: 3
SharedFileDesc structs used now: 1
===== LIBCx stats end =====
Assertion failed: arc == NO_ERROR, file D:/Users/dmik/rpmbuild/BUILD/libcx-0.7.0/src/shared.c, line 564

Killed by SIGABRT
pid=0x01b3 ppid=0x0010 tid=0x0001 slot=0x0065 pri=0x0200 mc=0x0001 ps=0x0010
X:\APACHE\BIN\HTTPD.EXE
Process dumping was disabled, use DUMPPROC / PROCDUMP to enable it.

emax

2021-06-22 03:14

reporter   ~0003751

update

i've upgraded to apache 2.4.48 let's see if this improve the situation, but i doubt

Server Version: Apache/2.4.48 (OS/2) OpenSSL/1.1.1k PHP/5.6.40
Server MPM: mpmt_os2
Server Built: Jun 6 2021 15:51:55

emax

2021-06-22 06:05

reporter   ~0003752

same issue even on 2.4.48

about the github repository it's allmost abandoneware there

psmedley

2021-06-22 06:59

administrator   ~0003753

It is no surprise that 2.4.48 behaves the same. As I've said, the 'issue' is coming from libcx.

Best I can do is attempt to build apache2 without a libcx dependency.

Not sure how feasible this is as libcx provides a number of functions that are considered 'standard' these days.

emax

2021-06-22 07:27

reporter   ~0003754

i don't know what to say, what i see is that the repository on github is abandoneware
and followed by anyone and nobody answer
that is no good at all
apache 2.4.x is too much unstable that i'm considering comin back to 2.2.x

psmedley

2021-06-22 09:35

administrator   ~0003755

You're welcome to switch back to Apache 2.2 (it's your system), just be aware that there will be no future builds/fixes for it, it's way too far EOL.

psmedley

2021-06-22 18:54

administrator   ~0003756

https://smedley.id.au/tmp/httpd-2.4.48-os2-20210622-debug.zip

*minimises* use of libcx to only ssl.dll and md.dll and doesn't use mmap/munmap at all.

Note: you won't get exceptq logs from this one....

emax

2021-06-22 19:04

reporter   ~0003757

even after upgrading from libcx 0.6.6 to 0.7.0 i don't have aymore exceptQ dumps

emax

2021-06-24 23:23

reporter   ~0003758

https://smedley.id.au/tmp/httpd-2.4.48-os2-20210622-debug.zip

This new build die on start:

A non-recoverable error occurred. The process ended.

06-24-2021 15:48:07 SYS2070
PID 0487 TID 0001 Slot 0085
X:\APACHE\BIN\HTTPD.EXE
AUTHN_CO->HTTPD._ap_hook_check_authn 127

no exceptQ dump at all
at this point i think that libcx 0.7.0 is not compatible with excetpQ environment

psmedley

2021-06-25 06:52

administrator   ~0003759

That looks like you have an old DLL in libpath. However, I realised there's more to it than just building apache2 without libcx, as php also uses it, and for mmap support, it's required that the executable that loads a DLL is also linked with libcx.

I did advise that there would be no exceptq support - the reason for that is quite simple. There are two ways to add exceptq support to an executable. 1) modify the code; 2) link against libcx. The apache2 code used to use approach 1), but when we started using libcx, we removed the code as it made patching easier. So in this build, there is no mechanism to load exceptq.

In your error above, there would never be a TRP file anyway, as there is no exception - it just can't load a DLL due to incompatible symbols.

Your issues with libcx 'assertions failed' also won't generate an exceptq dump, as there is no crashing code - libcx is deliberately aborted as it got something unexpected.

emax

2021-06-29 06:21

reporter   ~0003765

hi Paul,

i guess that there is no other way if Bitwise or Dmik put their hand on libcx 0.7.0...

But that repository on github seems like a desert :(

Steven Levine

2021-07-02 06:22

manager   ~0003774

Just a reminder. When I see a summary like:

  Apache 2.4.46 build (1th/4/2021) with php 5.6.40 sometimes don't restart

and 2.4.48 has been out for a while, I tend to ignore the the ticket assuming it a stale. If you have replicated the issue with 2.4.48, you need to take the time to update the ticket summary.

Steven Levine

2021-07-10 09:56

manager   ~0003791

If you can reproduce this with httpd-2.4.48-os2-20210619-debug.zip, please capture a process dump using http://www.warpcave.com/os2diags/PDumpCtl-0.17-20210620.zip. Since the hang is most likely in a DLL, you will need to capture a full dump using something like:

  pdumpctl httpd f d o

emax

2021-07-10 19:17

reporter   ~0003794

- pdumpctl environment updated

- now i'm going to update apache if it is not that version with reduced use of libcx, since here it neither start

- there is a big misunderstanding this is not an hang of httpD and i've an unkillable process, apache exit with that crash and don't start anymore
so that i've to setboot /b the VM, so how can i use dumpctl on httpd if i've no httpd running in memory?

thanks

emax

2021-07-10 19:26

reporter   ~0003796

this build httpd-2.4.48-os2-20210619-debug.zip. don't start

A non-recoverable error occurred. The process ended.

07-10-2021 11:53:50 SYS2070 PID 0208 TID 0001 Slot 0086
X:\APACHE\BIN\HTTPD.EXE
AUTHN_CO->HTTPD._ap_hook_check_authn
127

probably it lacks some modules

Steven Levine

2021-07-15 14:16

manager   ~0003818

Is this ticket ready to mark resolved?

emax

2021-07-15 23:55

reporter   ~0003821

i've a lot of this issue in these days, this ticket is maybe the most important
and the issue the most problematic

apache that don't restart anymore so that the script automatically after 3 retries give a setboot /b
i had this issue 3 times today

i'm starting to believe that apache 2.4.x is not jet compatible with a production environment

i did not have all these issue with 2.2.x
sorry

massimo

Steven Levine

2021-07-16 01:27

manager   ~0003823

You are free to continue to run 2.2. The only downside is I will not provide any support for 2.2. I only have so much time available. You will be on your own unless someone else decides to support 2.2.

If you choose to help get the 2.4 issues resolved, you need to collect the required data so that others have data to look at. You know which data is required because this is not the first time you have asked for help on similar issues. You reported similar issues regularly for 2.2.

The choice is yours.

emax

2021-07-26 06:19

reporter   ~0003837

note: upgraded apache to debug build Jun 19 2021 07:52:39

Issue History

Date Modified Username Field Change
2021-06-04 19:27 emax New Issue
2021-06-04 19:32 emax Note Added: 0003706
2021-06-04 19:33 psmedley Note Added: 0003707
2021-06-04 20:24 psmedley Note Added: 0003708
2021-06-04 20:25 psmedley Note Added: 0003709
2021-06-04 20:29 emax Note Added: 0003710
2021-06-04 20:40 emax Note Added: 0003711
2021-06-04 20:50 emax Note Added: 0003712
2021-06-05 07:25 psmedley Note Added: 0003713
2021-06-07 21:18 emax Note Added: 0003717
2021-06-14 18:01 emax Note Added: 0003733
2021-06-14 18:04 emax Note Added: 0003735
2021-06-21 23:38 emax Note Added: 0003750
2021-06-22 03:14 emax Note Added: 0003751
2021-06-22 06:05 emax Note Added: 0003752
2021-06-22 06:59 psmedley Note Added: 0003753
2021-06-22 07:27 emax Note Added: 0003754
2021-06-22 09:35 psmedley Note Added: 0003755
2021-06-22 18:54 psmedley Note Added: 0003756
2021-06-22 19:04 emax Note Added: 0003757
2021-06-24 23:23 emax Note Added: 0003758
2021-06-25 06:52 psmedley Note Added: 0003759
2021-06-29 06:21 emax Note Added: 0003765
2021-07-02 06:22 Steven Levine Note Added: 0003774
2021-07-10 09:56 Steven Levine Assigned To => Steven Levine
2021-07-10 09:56 Steven Levine Status new => feedback
2021-07-10 09:56 Steven Levine Note Added: 0003791
2021-07-10 19:17 emax Note Added: 0003794
2021-07-10 19:17 emax Status feedback => assigned
2021-07-10 19:26 emax Note Added: 0003796
2021-07-15 14:16 Steven Levine Status assigned => feedback
2021-07-15 14:16 Steven Levine Note Added: 0003818
2021-07-15 23:55 emax Note Added: 0003821
2021-07-15 23:55 emax Status feedback => assigned
2021-07-16 01:27 Steven Levine Note Added: 0003823
2021-07-26 06:19 emax Note Added: 0003837