View Issue Details
ID | Project | Category | View Status | Date Submitted | Last Update |
---|---|---|---|---|---|
0000592 | Squid for OS/2 | Bug | public | 2013-12-28 15:36 | 2022-09-24 02:50 |
Reporter | LewisR | Assigned To | psmedley | ||
Priority | normal | Severity | minor | Reproducibility | always |
Status | closed | Resolution | reopened | ||
Summary | 0000592: Squid 3.1.23 (squid-3.1.23-os2-20131228b) does not respect max filedescriptors config tag | ||||
Description | Squid 3.1 implements a new configuration tag: # TAG: max_filedescriptors # The maximum number of filedescriptors supported. # # The default "0" means Squid inherits the current ulimit setting. # # Note: Changing this requires a restart of Squid. Also # not all comm loops supports large values. #Default: # max_filedescriptors 0 Setting this either to a number lower than default (2048, apparently - ulimit?) results in no change to reported file descriptors, nor does setting a number higher than default change the reported descriptor count. | ||||
Steps To Reproduce | 1. Edit squid.conf to include the line: max_filedescriptors 1024 2. Restart squid Result (from log & vio): 2013/12/28 10:07:33| Open FD UNSTARTED 5 DNS Socket IPv4 2013/12/28 10:07:33| Open FD UNSTARTED 14 ICP Socket 2013/12/28 10:07:33| Open FD UNSTARTED 15 HTCP Socket 2013/12/28 10:07:33| Squid Cache (Version 3.1.23): Exiting normally. 2013/12/28 10:12:08| Starting Squid Cache version 3.1.23 for i386-pc-os2-emx... 2013/12/28 10:12:08| Process ID 240 2013/12/28 10:12:08| With 2048 file descriptors available 2013/12/28 10:12:08| Initializing IP Cache... Expected: 2013/12/28 10:07:33| Open FD UNSTARTED 5 DNS Socket IPv4 2013/12/28 10:07:33| Open FD UNSTARTED 14 ICP Socket 2013/12/28 10:07:33| Open FD UNSTARTED 15 HTCP Socket 2013/12/28 10:07:33| Squid Cache (Version 3.1.23): Exiting normally. 2013/12/28 10:12:08| Starting Squid Cache version 3.1.23 for i386-pc-os2-emx... 2013/12/28 10:12:08| Process ID 240 2013/12/28 10:12:08| With 1024 file descriptors available 2013/12/28 10:12:08| Initializing IP Cache... | ||||
Additional Information | Tried setting this to 65535; same result, except we get a warning that the count could not be changed: 2013/12/28 10:06:33| Starting Squid Cache version 3.1.23 for i386-pc-os2-emx... 2013/12/28 10:06:33| Process ID 235 2013/12/28 10:06:33| NOTICE: Could not increase the number of filedescriptors 2013/12/28 10:06:33| With 2048 file descriptors available 2013/12/28 10:06:33| Initializing IP Cache... squidclient mgr:info says (after several minutes of uptime, but no browsing): File descriptor usage for squid: Maximum number of file descriptors: 2048 Largest file desc currently in use: 30 Number of file desc currently in use: 18 Files queued for open: 0 Available number of file descriptors: 2030 Reserved number of file descriptors: 100 Store Disk files open: 0 How to set ulimit on OS/2, anyway? Current CONFIG settings on this machine: SHELLHANDLESINC=200 Not sure what else could/should/might need to be set. | ||||
Tags | No tags attached. | ||||
|
Sorry, Paul. Please edit summary to correct my typo: [...] does not respect max filedescriptors config tag ^^^ Tired, and not seeing straight (yet)! Thanks for the new build!! |
|
This behaviour will probably still be the same with 3.2 and 3.3 - let's confirm first then I will look at fixing. |
|
Please test with http://smedley.id.au/tmp/squid-3.3.11-os2-20131229.zip - I expect it will be the same |
|
OK pretty sure this will still fail. I will need to add OS2 specific code to SetMaxFD(). The good thing is that I think I may already have example code from apache2 :) Will try work something up this afternoon. About to head out dog walking then for lunch :P |
|
Hope you had a good walk and a good lunch. :-) Confirmed that this behavior persists with 3.3.11-os2-20131229. |
|
Interesting - from the squid code: #if USE_SELECT || USE_SELECT_WIN32 /* select() breaks if this gets set too big */ if (Config.max_filedescriptors > FD_SETSIZE) { rl.rlim_cur = FD_SETSIZE; debugs(50, DBG_CRITICAL, "WARNING: 'max_filedescriptors " << Config.max_filedescriptors << "' does not work with select()"); } else #endif FD_SETSIZE is set to 2048 in libc's sys/select.h |
|
Steven: interested in your thoughts on http://mantis.smedley.id.au/view.php?id=592#c2615 |
|
Note: from sys/select.h: /** Define the number of file handles in the select buffer. * @remark we might wanna bump this one up a bit... */ #if !defined (FD_SETSIZE) #define FD_SETSIZE 2048 #elif FD_SETSIZE < 256 #error FD_SETSIZE must be at least 256 #endif So perhaps 2048 is a fairly arbitrary value.... |
|
ok http://smedley.id.au/tmp/squid.zip is updated and built without -Zhigh-mem which fixes the 'bad address' error and also has some code in it to attempt to set higher numbers of file handles.... |
|
FS_SETSIZE is somewhat arbitrary and in our case a fixed value. On other platforms it can grow, IIRC. The only downside to the 2048 value is that it puts a limit on the maximum number of sockets that select can handle. On OS/2 where the socket handles and the file handles are in separate number spaces, 2048 should be fine except for the busiest of systems. On systems where the socket handles and file handles share the same number space, larger values of FD_SETSIZE might be required. I assume that you used DosSetRelMaxFH to bump the handle count when needed. This is a good solution. |
|
http://smedley.id.au/tmp/squid-3.3.11.diff has the current diffs. |
|
Using: 12-29-13 21:44 3,549,328 0 squid.exe with: max_filedescriptors 65535 squidclient.exe mgr:info reports: File descriptor usage for squid: Maximum number of file descriptors: 10000 Largest file desc currently in use: 66 Number of file desc currently in use: 10 Files queued for open: 0 Available number of file descriptors: 9990 Reserved number of file descriptors: 100 Store Disk files open: 0 So, when you mentioned in note #2619 that you added code to attempt to set higher numbers of file handles, I assumed you meant that you set a higher hard limit, however, I see from the diff that you have defined 4096 as the limit when max_filedescriptors is not set, and so my 10000 figure above may be imposed by my CONFIG.SYS setting of 200 files. I'll experiment with a couple of other settings and see what I get insofar as limiting is concerned. |
|
I more just changed to 4096 to see what would happen :0 I'm not sure where the 10000 value comes from - I didn't dig through the code long enough to find out. |
|
:-) The annotated config says the following regarding max_filedescriptors: # TAG: max_filedescriptors # Reduce the maximum number of filedescriptors supported below # the usual operating system defaults. # # Remove from squid.conf to inherit the current ulimit setting. # # Note: Changing this requires a restart of Squid. Also # not all I/O types supports large values (eg on Windows). #Default: # Use operating system limits set by ulimit. I just got done combing through my 3.0 stable 26 conf, migrating settings as needed to the new version (as with many of these apps, the available options and syntax can change so drastically between several versions that trying to merge the two files - or attempting to reuse the old one - is more trouble than just biting the bullet, scanning through, and perhaps cleaning out some cruft in the process). I took the opportunity to lower the setting to 5000 in squid.conf, before a complete shutdown & restart (of squid): max_filedescriptors 5000 No difference; I still have 10,000 file descriptors available. :-) I'm not complaining, as I thought that 65535 was extreme, anyway, and 10,000 should be more than adequate. I'll try trimming SHELLHANDLESINC before the next reboot to see if that imposes a limit. |
|
A couple of things have bubbled back up to the surface today; this was one of them... ;-) Lowering SHELLHANDLESINC to 50 makes no difference; we still have a hard limit of 10000 descriptors and the conf setting does not limit them. The startup log seems to reveal something, though: 2014/01/29 12:57:32 kid1| ERROR: setrlimit: RLIMIT_NOFILE: (78) Function not implemented 2014/01/29 12:57:32 kid1| ERROR: setrlimit: RLIMIT_NOFILE: (78) Function not implemented 2014/01/29 12:57:32 kid1| Starting Squid Cache version 3.3.11 for i386-pc-os2-emx... 2014/01/29 12:57:32 kid1| Process ID 98 2014/01/29 12:57:32 kid1| Process Roles: worker 2014/01/29 12:57:32 kid1| With 10000 file descriptors available 2014/01/29 12:57:32 kid1| Initializing IP Cache... 2014/01/29 12:57:32 kid1| DNS Socket created at 0.0.0.0, FD 7 2014/01/29 12:57:32 kid1| Adding domain randr from /@unixroot/etc/resolv.conf 2014/01/29 12:57:32 kid1| Adding nameserver 192.168.100.1 from /@unixroot/etc/resolv.conf 2014/01/29 12:57:32 kid1| Adding nameserver 4.2.2.2 from /@unixroot/etc/resolv.conf 2014/01/29 12:57:32 kid1| Adding nameserver 4.2.2.1 from /@unixroot/etc/resolv.conf 2014/01/29 12:57:33 kid1| Logfile: opening log daemon:/squid/var/logs/access.log 2014/01/29 12:57:33 kid1| Logfile Daemon: opening log /squid/var/logs/access.log 2014/01/29 12:57:33 kid1| Unlinkd pipe opened on FD 15 2014/01/29 12:57:33 kid1| Store logging disabled 2014/01/29 12:57:33 kid1| Swap maxSize 102400 + 262144 KB, estimated 28041 objects 2014/01/29 12:57:33 kid1| Target number of buckets: 1402 2014/01/29 12:57:33 kid1| Using 8192 Store buckets 2014/01/29 12:57:33 kid1| Max Mem size: 262144 KB 2014/01/29 12:57:33 kid1| Max Swap size: 102400 KB 2014/01/29 12:57:33 kid1| Rebuilding storage in /squid/var/cache (dirty log) 2014/01/29 12:57:33 kid1| Using Least Load store dir selection 2014/01/29 12:57:33 kid1| chdir: /squid/var/cache/squid: (2) No such file or directory 2014/01/29 12:57:33 kid1| Current Directory is C:/squid/sbin 2014/01/29 12:57:34 kid1| Loaded Icons. 2014/01/29 12:57:34 kid1| Configuring Parent localhost/8118/0 2014/01/29 12:57:34 kid1| Accepting HTTP Socket connections at local=0.0.0.0:3128 remote=[::] FD 18 flags=9 2014/01/29 12:57:34 kid1| Accepting ICP messages on 0.0.0.0:3130 2014/01/29 12:57:34 kid1| Sending ICP messages from 0.0.0.0:3130 2014/01/29 12:57:34 kid1| Accepting HTCP messages on 0.0.0.0:4827 2014/01/29 12:57:34 kid1| Sending HTCP messages from 0.0.0.0:4827 2014/01/29 12:57:34 kid1| Store rebuilding is 7.71% complete 2014/01/29 12:57:35 kid1| Done reading /squid/var/cache swaplog (51879 entries) 2014/01/29 12:57:35 kid1| Finished rebuilding storage from disk. 2014/01/29 12:57:35 kid1| 28181 Entries scanned 2014/01/29 12:57:35 kid1| 0 Invalid entries. 2014/01/29 12:57:35 kid1| 0 With invalid flags. 2014/01/29 12:57:35 kid1| 4483 Objects loaded. 2014/01/29 12:57:35 kid1| 0 Objects expired. 2014/01/29 12:57:35 kid1| 23698 Objects cancelled. 2014/01/29 12:57:35 kid1| 0 Duplicate URLs purged. 2014/01/29 12:57:35 kid1| 0 Swapfile clashes avoided. 2014/01/29 12:57:35 kid1| Took 2.01 seconds (2230.35 objects/sec). 2014/01/29 12:57:35 kid1| Beginning Validation Procedure 2014/01/29 12:57:37 kid1| Completed Validation Procedure 2014/01/29 12:57:37 kid1| Validated 4483 Entries 2014/01/29 12:57:37 kid1| store_swap_size = 92156.00 KB 2014/01/29 12:57:37 kid1| storeLateRelease: released 0 objects I think the errors mentioned here correspond to some the changes you implemented, Paul, as indicated in your diff. |
|
FYI: 2014/01/29 12:57:33 kid1| chdir: /squid/var/cache/squid: (2) No such file or directory is a valid message. I had a misconfiguration pointing to the core dump directory as the default (stated above). Indeed, there is no squid directory under cache/. Fixed. |
|
Please test with https://smedley.id.au/tmp/squid-5.3-os2-20220118.zip |
|
I must have missed the email from January with your last comment. Unfortunately, Squid 5.3 fails to start for me. I'll open a separate ticket, and wil close this one again (just wanted to note my experience, here). |
Date Modified | Username | Field | Change |
---|---|---|---|
2013-12-28 15:36 | LewisR | New Issue | |
2013-12-28 15:39 | LewisR | Note Added: 0002607 | |
2013-12-28 21:38 | psmedley | Note Added: 0002609 | |
2013-12-28 21:53 | psmedley | Note Added: 0002611 | |
2013-12-28 22:23 | psmedley | Note Added: 0002612 | |
2013-12-28 22:24 | psmedley | Summary | Squid 3.1.23 (squid-3.1.23-os2-20131228b) does not respect mx filedescriptors config tag => Squid 3.1.23 (squid-3.1.23-os2-20131228b) does not respect max filedescriptors config tag |
2013-12-29 00:35 | LewisR | Note Added: 0002613 | |
2013-12-29 05:07 | psmedley | Note Added: 0002615 | |
2013-12-29 05:25 | psmedley | Note Added: 0002617 | |
2013-12-29 05:27 | psmedley | Note Added: 0002618 | |
2013-12-29 06:00 | psmedley | Note Added: 0002619 | |
2013-12-29 06:28 | Steven Levine | Note Added: 0002620 | |
2013-12-29 06:29 | Steven Levine | Note Edited: 0002620 | |
2013-12-29 08:00 | psmedley | Note Added: 0002621 | |
2013-12-30 20:21 | LewisR | Note Added: 0002634 | |
2013-12-31 06:12 | psmedley | Note Added: 0002637 | |
2013-12-31 06:22 | LewisR | Note Added: 0002638 | |
2014-01-29 18:14 | LewisR | Note Added: 0002650 | |
2014-01-29 18:24 | LewisR | Note Added: 0002652 | |
2022-09-24 01:22 | psmedley | Assigned To | => psmedley |
2022-09-24 01:22 | psmedley | Status | new => resolved |
2022-09-24 01:22 | psmedley | Resolution | open => no change required |
2022-09-24 01:22 | psmedley | Note Added: 0004580 | |
2022-09-24 02:50 | LewisR | Status | resolved => feedback |
2022-09-24 02:50 | LewisR | Resolution | no change required => reopened |
2022-09-24 02:50 | LewisR | Note Added: 0004585 | |
2022-09-24 02:50 | LewisR | Status | feedback => closed |