View Issue Details

IDProjectCategoryView StatusLast Update
0000705Other Unix PortBugpublic2020-03-25 20:03
ReporteremaxAssigned ToSteven Levine 
PrioritynormalSeveritycrashReproducibilitysometimes
Status assignedResolutionopen 
Summary0000705: Stunnel 5.50 often crash
Descriptionstunnel 5.50 on i386-pc-os2-emx built by Paul Smedley on Jan 13 2019
Compiled/running with OpenSSL 1.0.2n 7 Dec 2017

often crash, exceptq dump in attachment

OS: eCS 2.1 + latest AN driver pack (installed on bare metal)
Server: SuperMicro Intel Xeon 4 core 2,4Ghz, 4GB ram

stunnel only used for pop3S (since weasel don't still have popS protocol), mta is on the same server (localhost)
Steps To Reproduce
dunno how to reproduce it, the feeiling is that it crash sometimes when mail(popS) clients go into timeout or such

stunnel 5.50 also quite often charge the CPUs with high load (50%), the OS uses ACPI and see 4 cores

TagsNo tags attached.

Relationships

has duplicate 0000710 closedSteven Levine Stunnel 5.50 often crash (ticket ONLY for the crash issue) 

Activities

emax

2020-02-29 21:14

reporter  

stunnel_crash.zip (5,568 bytes)
configuration.zip (622 bytes)

Steven Levine

2020-03-03 12:57

manager   ~0003387

Initial analysis indicates there is a buffer overflow in on of the libcrypo routines. The .trp file reports:

 Trap -> 00189587 STUNNEL 0001:00179587 between _sha1_block_data_order + 19D7 and _sha256_block_data_order - 1669 (in sha1-586.obj and sha256-586.obj)

However the large offset implies to the trap itself in in a supporting routine rather than _sha1_block_data_order.

I need to be a bit more analysis that better understand whether this is a error fixed by a newer libcrypo build or if this is a compiler issue.

When uploading small text files, please do not zip them before uploading. It makes extra work for me. Please delete the zip files and upload the original text files.


Steven Levine

2020-03-03 14:48

manager   ~0003388

Please check an verify that your stunnel.xqs matches the installed stunnel. I am getting some results that imply it is not.

emax

2020-03-03 20:54

reporter   ~0003389

this archive: http://smedley.id.au/downloads/stunnel-5.50-os2.zip
has no XQS :(

Steven Levine

2020-03-04 04:00

manager   ~0003390

To refresh your memory, none of Paul's stunel builds include .xqs files. Use mapxqs.exe to convert the supplied map files to a .xqs files. You have done this before. Please do not force me to have to remind you again or I will eventually lose patience. Since you appear to tend to forget these kinds things, I recommend you assemble a procedures notebook for the tasks you don't do regularly. I find this helps me avoid silly mistakes and wasting other peoples time.

Also, please convert the .zip file uploads as requested. I should not have a make this request more than once.

Finally, since the exceptq report you uploaded is mostly invalid, please capture a new .trp file and uplod it once you have the correct .xqs file in place.

emax

2020-03-04 10:19

reporter   ~0003391

attachment re-uploaded not zipped
updated .xqs file

i will upload the new exceptq dump file when it will be produced again

thanks

massimo

stunnel.conf (892 bytes)
062F_03.TRP (20,871 bytes)

emax

2020-03-05 19:09

reporter   ~0003393

i'm waiting for a new exceptq dump due to a crash

but in theese days i'm seeing often 100% cpu load from stunnel, is there any way i can help to understand why this happens?

emax

2020-03-05 19:19

reporter   ~0003394

i've a suggestion is it possible to have the latest version 5.56 compiled, maybe this is some kind of bug fixed in the newer versions?
so that at least we can check with the latest one (5.56)

thanks

massimo

emax

2020-03-06 08:34

reporter   ~0003396

hi Steven,

about stunnel 5.56 i've wrote an email to Paul

i've downloaded the 2 utilities and wrote scripts to easy run them in moments of needs (when cpu load is high)
anyway when the cpu load is high i can still run the command prompt without too much issues
and i've console access to the server

thanks

Steven Levine

2020-03-10 14:40

manager   ~0003409

This ticket is now reserved to stunnel crash analysis. We will use ticket #709 to work on the CPU utilization problem.

Steven Levine

2020-03-10 14:47

manager   ~0003410

Based on a quick analysis of the trap file, It appears we have a heap buffer overflow for some as yet unknown reason. More analysis to follow.

emax

2020-03-10 21:27

reporter  

004A_03.TRP (40,559 bytes)

Steven Levine

2020-03-11 06:34

manager   ~0003416

Once you haver verified that Paul's 5.56 build works as least as well as the 5.50 build, if you happen to get a trap, please upload the .trp file. The stunnel.exe in stunnel-5.56-debug.zip is built with debug data, so you do not need to install an updated .xqs file.

emax

2020-03-11 07:33

reporter   ~0003417

http://smedley.id.au/tmp/stunnel-5.56-os2.zip

i've this archive and there is no XQS inside of it

emax

2020-03-11 07:34

reporter   ~0003418

i've manually created the XQS file with mapxqs tool

Steven Levine

2020-03-11 10:04

manager   ~0003419

Which part of https://mantis.smedley.id.au/view.php?id=705#c3416 did you not understand?

Also, please review your procedures notebook where you wrote down that Paul's distributions never include .xqs files.

emax

2020-03-11 18:45

reporter   ~0003420

Steven, you talk about this archive: stunnel-5.56-debug.zip

i only have this one: http://smedley.id.au/tmp/stunnel-5.56-os2.zip

about the .xqs file i've manually created it with mapxqs, is this ok for the ticket?
yes or not ?

psmedley

2020-03-12 19:00

administrator   ~0003421

As Steven said, if you're running a debug build, you don't need the XQS file.... (per "The stunnel.exe in stunnel-5.56-debug.zip is built with debug data, so you do not need to install an updated .xqs file."

emax

2020-03-12 19:33

reporter   ~0003422

ok, now i've undestand

but i run this http://smedley.id.au/tmp/stunnel-5.56-os2.zip
and i've produced manually with mapxqs the xqs files
is this ok or not?

massimo

emax

2020-03-13 02:44

reporter  

004A_03-2.TRP (60,854 bytes)

Steven Levine

2020-03-13 09:14

manager   ~0003424

Massimo, what you have done so far is fine.

Do you now understand the relationship between debug data and .xqs files? This is covered in gory detail in exceptq.txt and exceptq-shl.txt.

What we want you to do going forward is to use

  https://smedley.id.au/tmp/stunnel-5.56-debug.zip

This is built from the same sources as:

 http://smedley.id.au/tmp/stunnel-5.56-os2.zip

but because of how compilers work, the build are not exactly the same. It is much more efficient to limit my work to a few builds as possible.

You did not get the link for the debug build because it did not exist when Paul provided you the link to the first 5.56 build.

The .trp file indicates that the 5.56 sources do not resolve this ticket's issue. Once you have 5.56 debug installed and can provide an updated .trp file, I will resume work on this issue.

emax

2020-03-13 09:23

reporter   ~0003426

thanks

updated stunnel to the debug build: https://smedley.id.au/tmp/stunnel-5.56-debug.zip

emax

2020-03-16 08:34

reporter   ~0003434

in attachment an stunnel crash ( build https://smedley.id.au/tmp/stunnel-5.56-debug.zip )

004A_05.TRP (16,524 bytes)

emax

2020-03-24 19:58

reporter   ~0003440

stunnel crash produced a new dump on 24 march 2020

004A_05-2.TRP (16,028 bytes)

Steven Levine

2020-03-25 02:47

manager   ~0003441

Please, do not upload any more .trp files until I have time to work through what we have. We have enough .trp files to know that the the same exception is occuring every time.

Steven Levine

2020-03-25 14:48

manager   ~0003442

The attached patch file should avoid the trap in _sha1_block_data_order()

The patch file has UNIX line endings. If the sources to be patched have DOS line ending, use unix2dos to convert the patch file's line endings.

md_rand.c.diff (1,144 bytes)
diff --git a/crypto/rand/md_rand.c b/crypto/rand/md_rand.c
index 2983a3f..317eb9c 100644
--- a/crypto/rand/md_rand.c
+++ b/crypto/rand/md_rand.c
@@ -271,6 +271,20 @@ static void ssleay_rand_add(const void *buf, int num, double add)
             goto err;
         k = (st_idx + j) - STATE_SIZE;
         if (k > 0) {
+            /* Smedley mantis ticket #705
+              2020-03-24 SHL
+              Under some conditions k > j and we trap.
+              This is possibly a code defect which results in
+              ssleay_rand_cleanup not getting called under some
+              conditions.
+              It is also possible that this is SMP locking issue which
+              allows some other thread to modify state_index with the result
+              that st_idx is initiailized to a larger than expected value.
+              To avoid the trap we return an error when this occurs
+              which is better than trapping
+            */
+            if (k > j)
+                goto err;
             if (!MD_Update(&m, &(state[st_idx]), j - k) ||
                 !MD_Update(&m, &(state[0]), k))
                 goto err;
md_rand.c.diff (1,144 bytes)

psmedley

2020-03-25 15:24

administrator   ~0003443

Thanks Steven, I'll aim to build tonight. Thinking out loud, whilst we have no direct evidence it's the case, I can't help but wonder if this bug causes issues for Apache/PHP as well, given both also use openssl.

I'll also rebuild those if I get time.

psmedley

2020-03-25 17:49

administrator   ~0003444

http://smedley.id.au/tmp/stunnel-5.56-debug-20200325.zip

emax

2020-03-25 20:03

reporter   ~0003446

25/3/2020 10,30 AM upgraded stunnel to http://smedley.id.au/tmp/stunnel-5.56-debug-20200325.zip

Issue History

Date Modified Username Field Change
2020-02-29 21:14 emax New Issue
2020-02-29 21:14 emax File Added: stunnel_crash.zip
2020-02-29 21:14 emax File Added: configuration.zip
2020-03-03 09:22 Steven Levine Assigned To => Steven Levine
2020-03-03 09:22 Steven Levine Status new => assigned
2020-03-03 12:57 Steven Levine Status assigned => feedback
2020-03-03 12:57 Steven Levine Note Added: 0003387
2020-03-03 14:48 Steven Levine Note Added: 0003388
2020-03-03 20:54 emax Note Added: 0003389
2020-03-03 20:54 emax Status feedback => assigned
2020-03-04 04:00 Steven Levine Status assigned => feedback
2020-03-04 04:00 Steven Levine Note Added: 0003390
2020-03-04 10:19 emax File Added: stunnel.conf
2020-03-04 10:19 emax File Added: 062F_03.TRP
2020-03-04 10:19 emax Note Added: 0003391
2020-03-04 10:19 emax Status feedback => assigned
2020-03-05 19:09 emax Note Added: 0003393
2020-03-05 19:19 emax Note Added: 0003394
2020-03-06 06:20 Steven Levine Status assigned => feedback
2020-03-06 08:34 emax Note Added: 0003396
2020-03-06 08:34 emax Status feedback => assigned
2020-03-06 19:33 emax File Added: 100percent_out.txt
2020-03-06 19:33 emax File Added: top.jpg
2020-03-06 19:33 emax File Added: PDUMP_with_high_load_cpu.zip
2020-03-10 14:22 Steven Levine Issue cloned: 0000709
2020-03-10 14:40 Steven Levine Note Added: 0003409
2020-03-10 14:41 Steven Levine File Deleted: top.jpg
2020-03-10 14:43 Steven Levine File Deleted: 100percent_out.txt
2020-03-10 14:45 Steven Levine File Deleted: PDUMP_with_high_load_cpu.zip
2020-03-10 14:47 Steven Levine Note Added: 0003410
2020-03-10 21:27 emax File Added: 004A_03.TRP
2020-03-11 02:28 Steven Levine Relationship added has duplicate 0000710
2020-03-11 06:34 Steven Levine Status assigned => feedback
2020-03-11 06:34 Steven Levine Note Added: 0003416
2020-03-11 07:33 emax Note Added: 0003417
2020-03-11 07:33 emax Status feedback => assigned
2020-03-11 07:34 emax Note Added: 0003418
2020-03-11 10:04 Steven Levine Status assigned => feedback
2020-03-11 10:04 Steven Levine Note Added: 0003419
2020-03-11 18:45 emax Note Added: 0003420
2020-03-11 18:45 emax Status feedback => assigned
2020-03-12 19:00 psmedley Note Added: 0003421
2020-03-12 19:33 emax Note Added: 0003422
2020-03-13 02:44 emax File Added: 004A_03-2.TRP
2020-03-13 09:14 Steven Levine Status assigned => feedback
2020-03-13 09:14 Steven Levine Note Added: 0003424
2020-03-13 09:23 emax Note Added: 0003426
2020-03-13 09:23 emax Status feedback => assigned
2020-03-16 08:34 emax File Added: 004A_05.TRP
2020-03-16 08:34 emax Note Added: 0003434
2020-03-24 19:58 emax File Added: 004A_05-2.TRP
2020-03-24 19:58 emax Note Added: 0003440
2020-03-25 02:47 Steven Levine Note Added: 0003441
2020-03-25 14:48 Steven Levine File Added: md_rand.c.diff
2020-03-25 14:48 Steven Levine Note Added: 0003442
2020-03-25 15:24 psmedley Note Added: 0003443
2020-03-25 17:49 psmedley Note Added: 0003444
2020-03-25 20:03 emax Note Added: 0003446