View Issue Details

IDProjectCategoryView StatusLast Update
0000637Apache 2.x Bugpublic2019-09-29 10:41
ReporterLewisRAssigned To 
PrioritynormalSeveritymajorReproducibilitysometimes
Status newResolutionopen 
Product Version 
Target VersionFixed in Version 
Summary0000637: Apache 2.2.29 / PHP 5.4.34 issues - server: eCSVirtual
DescriptionThis is a metabug for issues discovered/investigated running Apache 2.2.29 / PHP 5.4.34 under VirtualBox eCS 2.1 VM (eCSVirtual) hosted on openSUSE 13.1 x64.
Additional InformationThis chronicles progress which started with an email thread dated 10/30/2014:

Well, as we've been going through another round of "where's the server?"
over here, I spent the entire day yesterday working with Apache in a VM.

I can replicate the stall condition, and I can even make the VM lock up
hard and even reboot. All of this can be rather easily controlled via
manipulating the MPM settings. Some examples:

Set the threadstacksize too low (65535), and Apache will blow the stack
and restart itself. This isn't an immediately-fatal problem for the
server, of course, but there is a time penalty for Apache to restart. To
get an idea of the time it takes for Apache to get back on its feet, run
sid2 -ig a few times while Apache is starting up; you can watch the
thread count rise as the daemon is initializing. Also, after a few
rounds of this, the server becomes unstable and usually reboots.

Set the threadstacksize too high (over 524280, for example), and the
server can - and will - consume all RAM and either hang or reboot.

Set the minsparethreads too low (10), and the performance is dreadful,
as we have to keep spawning new threads as the server gets busy. Too
high, and we can run out of threads (need to watch the arithmetic,
mindful of CONFIG.SYS limit, minsparethreads, maxsparethreads, and
startservers - threads are per-server).

Siege tells us that with PHP not loaded (as module or CGI), we get 100%
hits (using three parallel sieges of 200 semi-random urls - i.e., three
sets of urls from three different vhosts, all running PHP-based sites).
The expected response, of course, is Apache advising that the script
could not be executed. However, the net effect on Apache is that it
handles the request and returns a result - every time. Apache does
consume some memory during this exercise, but not nearly as much as when
PHP is loaded (as would be expected).

Loading PHP as module results in hits approximately 30-35% of the time,
using the same URLs against the same VM, with all other Apache
configuration the same. The important thing to note here is that this
same hit % is seen with and without the MySQL module loaded. I was going
to blame slow MySQL responses, until I saw that the response times in
the siege log were nearly identical. PHP should have simply returned an
error that the MySQL module was not loaded, and this should have been
very quick. Still, this is not the crux of the resource usage issue.

Running as module, it is not uncommon to see Apache (and modules)
continue to consume (*consume*, not just "not release") RAM *after* the
siege is over, and the server should be idle. More on this, below.

The PHP docs recommend a pre-fork server vs threaded when running PHP as
module:

http://php.net/manual/en/faq.installation.php#faq.installation.apache2

So, I reconfigured for CGI...

Siege response is a little less than it is when running PHP as module
(20-25%), however, what this did reveal was the following:

 1. Apache does not get busy.
 2. Apache uses only slightly more RAM than when sitting idle.
 3. *Spawned PHP.EXE processes either take a long time to exit or do not
    exit at all, and continue to consume more memory as they linger.*

So, run a siege with 200 clients, and a mess of PHP sessions are
spawned. Unfortunately, even after the siege is over, they never go
away. The majority of them may be killed manually (I had about a dozen
left over which would not kill). Unfortunately, running a second siege
compounds the problem, leaving us with another batch of PHP.EXEs running.

Apache becomes unkillable, and memory continues to be consumed. I
started with about 200MB RAM in use in a 2GB VM with a 2MB swap. About
two minutes after the siege ended, I was using about 1GB (not unusual),
but then it steadily grew until I had consumed all of the 2GB and the
swap file grew to almost 300MB before I started killing off PHP
sessions. Ultimately, I got back down to about 300MB in use, but the
last dozen PHP sessions would not kill and neither would Apache (though
I was able to remove it from the exit list).

I confirmed this same behavior on the 2014-06-10 builds of Apache 2.2.27
and PHP 5.3.28. And it is *not* the MySQL module (or any other dynamic
module), as I was able to replicate the condition with *all* modules
commented out of php.ini. (That doesn't mean that none of the modules
leak, of course; it just means that *this* issue is *not* due to a leaky
dynamic module.)

My conclusion at this point is that PHP is indeed the culprit and not
Apache (threaded or forked, I think we're going to see the same issue
until we get PHP under control).

Tomorrow morning (10:40 EDT), we have an appointment with the oncologist
to go over the lab results and the ultrasound. After that, my plan is to
come back here, confirm this same behavior (or not) running PHP 5.2.17,
then roll forward to the latest builds of both again to get some
procdumps of a few of the PHP sessions and the stuck Apache session (the
children exit when httpd -k restart is called, but not the parent), and
I'll follow up with an ftp link to the files for us to share. Steve,
I'll be sure to zip up the conf directory tree for you, as well.

It's also interesting to bear in mind that this is a VM configured for
one CPU, so while the SMP kernel with ACPI is being loaded (ACPI
3.22.03), there is still only one CPU in use.
TagsNo tags attached.

Relationships

related to 0000638 assignedSteven Levine httpd mpm does not support MaxRequestsPerChild properly 

Activities

LewisR

2014-12-20 09:01

developer   ~0002940

Steve followed up:

After some tweaking of config.sys to optimized what is getting loaded, we
did a siege run. The result was a number of apparently stuck php.exe
instances. Lewis captured a "full" dump with pdumpctl of one of the stuck
instances. The dump shows that php.exe is stuck trying to write to
stdout.

At this point I am assuming that stdout is suppposed to be connected to a
httpd thread and that the thread is not reading the pipe for an as yet
unknown reason.

What is a bit confusing is the content of the write buffer. It does not
look right. I expected to see CGI output which tends to be text and what
I see is binary.

# %kx
005b:1e899864 00000001 0055a72c 00009e88 0012e7b0 [LIBC066 __write + 64]
005b:1e8c63ac 00000001 0055a72c 00009e88 03000000 [LIBC066 _std_write +
12c] 005b:00011067 00000001 0055a72c 00009e88 1d38b2df
005b:1d38c7b2 0055a72c 00009e88 00164160 00000000 [_php_output_write +
149] 005b:1d370120 00550898 00009e88 00164160 00478b14 [_tsrm_realpath +
29C] 005b:1d3f48ed 00550898 00009e88 0012e90c 1d36cb55
[_zend_print_zval_ex + 51] 005b:1d3f4944 1d3700f0 00478b14 00000000
1d470979 [_zend_print_zval + 21] 005b:1d3f1e52 00478b14 00000000 001906b8
0018f654 [_zend_print_variable + 19] 005b:1d441d9e 00478b14 0016da14
00191c28 00164160 [_execute + 2216A] 005b:1d41fe4b 0016dc14 00000000
00000004 1d401623 [_execute + 217] 005b:1d3f4ed1 0018a238 00164160
0000004b 0012ea3c [_zend_execute_scripts + 7A] 005b:1d375158 00000008
00164160 00000000 00000003 [_php_execute_script + 1E8] 005b:000148b0
0012fe6c 00164160 00000000 00000000
005b:00010021 00000001 0012ff9c 20030180 ffffffff
005b:1e88887b 00000000 1ffece38 0000115e 00000000 [LIBC066 __init_app + b]

This says we are stuck at write(1, 0x55a72c, 0x9e88) and call gated to

 %1ffc6f84 DOSCALL1 DOS32IWRITE

The buffer begins

# dd 0055a72c
0053:0055a72c 00088b1f 00000000 fdd50600 48db7aeb
0053:0055a73c ff0c2d92 682afafe 69176f4e 27d1d441
0053:0055a74c 6cb714ca b69a55cb 7725b72d d2a4b775
0053:0055a75c c4a09103 08b00932 35128752 15aefed7
0053:0055a76c 80489991 9eeaa4a0 3dd9ef99 24011653
0053:0055a77c c8ce7912 fe2b1588 9dc69bd4 2b893f49
0053:0055a78c 5be4d1e7 a20f8d1a 9f1fa4c9 ace0c2d7
0053:0055a79c 8e197b7e 9e169d2b de060cec df56b4c3

LewisR

2014-12-20 09:09

developer   ~0002941

Paul:

one clue is I found a setmode(fd, O_BINARY) which is OS2 specific in main\streams\plain_wrapper.c

A build of php 5.4.34 php5.dll that removes this is at http://smedley.id.au/tmp/php-5.4.34-dll-20141031.zip

Steve:

I guess it's worth a test, although I don't expect much difference.
There's a slight performance boost when running O_BINARY because less data
is received and sent. Where possible, I tend to run O_BINARY unless the
handle is a console handle.

Paul:

http://smedley.id.au/tmp/php-5.4.34-20141101.zip is a full rebuild which also contains map files for cli and cgi php.exe

Lewis:

The new php5.4.34 build seemed to lock up on me to the point where
I could not even get a dump file.

Paul:

http://smedley.id.au/tmp/php-5.4.34-20141101b.zip reverts the os2 specific implementation of socket non-blocking....

Lewis:

pdf.dll needs to be recompiled for the difference in the API with
this new PHP build.

Paul:

I would have expected the PHP 5.4 build of pdf.dll to work for all 5.4.x versions

Anyhow, http://smedley.id.au/tmp/pdf.zip contains a rebuild

Lewis:

I have *no* idea where any of these things may lead us, but the issue is
decidedly *worse* with the 20141101b build (got stuck twice, unable to
dump).

Paul:

I've reverted the removal of setmode() in the following - which should get us back to the same state as the original build of 5.4.34
http://smedley.id.au/tmp/php-5.4.34-20141101c.zip

LewisR

2014-12-20 09:15

developer   ~0002942

Last edited: 2014-12-20 09:20

View 2 revisions

Steve:

When you get to the state where the php instances are hung let's see
if we can figure out how the pipes are associated.. Run Theseus2 and use
Process->General Process Information and look at the open files for one of
the stuck instances. Note the system handle numbers for process handles
0, 1 and 2. Then view the process information for the httpd processes and
figure out which process (if any) has the other side of the pipe open. If
you have trouble with this, save the process information windows and the
system file list window to files and let me take a look. This is somewhat
non-trivial since each side of the pipe has a different system file handle
so it's a process of elimination.

[...]

Looking at Theseus2 output for php-cgi we have:

  Process System Flags File name // Flags...
     0000 0241 00 named pipe //
     0001 01AF 00 named pipe //
     0002 0242 00 named pipe //
     0007 005D 00 named pipe //

and for httpd we have

  Process System Flags File name // Flags...
     0000 004F 08 \DEV\CON // console
     0001 004F 08 \DEV\CON // console
     0002 0120 00 J:\APPS\APACHE2\LOGS\ERROR_LOG //
     0007 005D 00 named pipe //

I need to think about this a bit, but it appears reasonable. I think that
the php process should have closed handle 7, but this is not related to
the problem at hand. I need to figure out how to extract the pipe name
from the dump data.

[...]

The contents are as expected. The process is hung at the DosWrite. I
haven't yet figured out how to derive the pipe name from the handle. It
might be easier to add some debug code to httpd to log when it kicks off a
CGI process. I need to poke about in the source code...

---

We decided to use OS2TRACE to attempt to determine which pipes were not getting connected. See attached testing-http-php-with-os2trace-20141108.txt for procedures.

LewisR

2014-12-20 09:19

developer  

testing-http-php-with-os2trace-20141108.txt (8,431 bytes)
How to use Dave Blaschke's os2trace to test apache httpd and php (cgi)

2014-11-07 SHL Baseline
2014-11-07 SHL Rework to use alternate ServerRoot directory.
2014-11-08 SHL Document logging level
2014-11-08 SHL Preset DOSCALLS APIs selections.

== Prerequisites ==

  - 4OS2
  - Dave Blaschke's os2trace package.
    Available from http://www.angelfire.com/tx4/blaschke/OS2Trace/
  - My os2traceenv.cmd script.
    Available from www.warpcave.com/betas
  - A working apache2 setup.
  - A working php CGI setup.

== Files that will be modified ==

  Os2trace modifies executables to insert forwarders to the
  tracing routines.

  The following files will need to be modified

    httpd.dll
    php.exe (cgi version)
    php5.dll
    libc066.dll

== Install os2traceenv.cmd script ==

  - Install os2traceenv.cmd to a directory in the PATH.

== Install os2trace ==

  - Create a directory for os2trace and unzip the distribution.
    I use d:\devtools\os2trace, but you can use a directory of
    your choice.

== Setup a mirror of your production apache2/php installation ==

  - Mirror your apache2 ServerRoot tree to a work directory.
    This makes setup simplifier and leaves the production apache's
    setup unsullied.
    I used d:\Internet\apache22-trc as the root, but you can use
    a directory of your choice.

  - Copy php5.dll to the mirror's bin directory.
    This makes BEGINLIBPATH setup simpler.

  - If your production cgi-bin is not apache2\cgi-bin, create
    a cgi-bin directory in the mirror and copy content of the
    production cgi-bin to the mirror's cgi-bin.

  - Edit the mirror's httpd.conf to match the new directory locations.
    Update ServerRoot to point to the mirror.
    Update ScriptAlias to point to the mirror's cgi-bin.
    Update absolute paths that should not point at production directories.
    Update log paths to leave the production logs unsullied.
    Update Directory sections as needed.
    Etc.

  - Copy libc066 DLLs to bin directory.
    This makes BEGINLIBPATH setup simpler.

  - Open a 4OS2 VIO session

  - Change to your os2trace directory and run

  - Start apache from the bin directory with

      httpd -d..

    to test your edits.

  - Apache should start a serve content.

  - Stop apache.

  - Do not close the VIO session.  You will continue to need it.

== Modify files for os2trace ==

  - Change to your os2trace directory and run

      os2traceenv

    to set up the os2trace environment.
    os2traceenv updates PATH and BEGINLIBPATH.

  - Change back to your mirror's bin directory

  - Run pmos2trc

  - Set the logging level to level 3

    From the Logging Level dialog, select the Log API parameter 
    contents (level 3) information item.

    Click OK to confirm the settings.

    The settings are stored in os2.ini in the os2trace 
    application and will persist until changed.

  - Customize the DOSCALLS APIs selections.

    Use the Customize -> DOSCALLS APIs menu option to tune the
    settings.

    From the DOSCALLS APIs dialog, turn on FILE PIPE and PROC and 
    turn off all the other selections.

    Click OK to confirm the settings.

    The settings are stored in os2.ini in the os2trace 
    application and will persist until changed.

  - Enable tracing for

      httpd.dll
      php.exe (cgi version)
      php5.dll
      libc066.dll

    For each file, use the Enable -> Open file menu option to
    select the file to modify.

    From the Enablement dialog turn on DOSCALLS and disable all 
    others Click OK to confirm the settings.

  - After all the files have been enabled for tracing, close
    pmos2trc leaving the VIO session open.

== Adjust PATHs ==

  - Edit BEGINLIBPATH and the mirror's bin directory so that 
    apache2 and php can find the modified php5.dll and the 
    modified libc066.dll when running LIBPATHSTRICT.

  - On my setup this results in

    BEGINLIBPATH=d:\internet\apache22-trc\bin;d:\devtools\os2trace;

  - Run

      set LIBPATHSTRICT=T

    to ensure that the modified libc066.dll is used.

== Start apache and verify os2trace operation ==

  - Start apache from the mirror's bin directory with

      httpd -d..

  - If all goes well, apache2 will start and os2trace will
    report that it generated unique .trc file names for the 2nd
    and 3rd httpd.exe instances.

  - Try a simple php test.

  - Verify that os2trace creates a php.trc file in the 
    mirror's cgi-bin directory.

  - Review the content of the generated .trc files.
    You should be able to see the pipe IO and file IO
    operations.
    You should also be able to see the CGI data that was read
    and written.

== Testing with siege ==

  - When running siege, os2trace will generate unique .trc file
    names for secondary php.exe instances.  The primary php.trc
    file will contain reports for the php.exe instances that 
    happened to be started when no other php.exe instances were 
    active.

  - The size of the .trc files will depend on the amount of IO
    done by the PHP code.

  - It is recommended that you turn off as much logging as
    possible.  This will reduce the amount of IO logged to the
    .trc files.  The siege test should not need any CustomLogs
    and LogLevel Error should be sufficient for our purposes.

  - The httpd.trc file can get large, as can the php.trc 
    file because they will contain the output for multiple siege 
    transactions.

  - A siege run will create a large number of php .trc files
    unless the siege dies quickly.  Each GET request for PHP 
    content will spawn a new PHP instance.

  - It's probably a good idea to stop httpd between siege runs
    and clean up any left over .trc files.  If a siege run fails,
    the .trc files should be moved to uniquely name archive
    directory for post-mortem analysis.  Otherwise, the .trc
    files can be deleted.

== Post mortem analysis ==

  - Once a siege run produces some hung php.exe instances, we are
    ready to analyze.

  - Find the php .trc files that are missing the final "stopping
    at" message.  These correspond to the hung instances.
    Verify that the PIDs match what

      pstat /c | grep PHP

    reports.

  - For each hung php instance, examine the httpd .trc files and
    find the Dos32ExecPgm invocation that started this php
    instances.

  - Note the PID/TID combo and search for API failures or
    unexpected operations on this thread.

  - more later ...

============================================================
====  Old setup method using saved copies of executables ===
============================================================

== Save copies to files that will be modified by os2trace ==

  - You will be instrumenting
      httpd.dll
      php.exe (cgi version)
      php5.dll
      libc066.dll

  os2trace can remove the modification, but I prefer to restore saved copies

== Modify files for os2trace ==

  -  Open a 4OS2 VIO session

  - Change to your os2trace directory and run

      os2traceenv

    to set up an os2trace environment.
    os2traceenv updates PATH and BEGINLIBPATH.

  - Run pmos2trc and enable tracing for

      httpd.dll
      php.exe (cgi version)
      php5.dll
      libc066.dll

    Use Enable -> Open file to select the file to modify
    In the Enablement dialog turn on DOSCALLS and disable all others
    Click OK to confirm the settings.

    Use Customize -> DOSCALLS APIs to tune the settings
    In the DOSCALLS APIs dialog, turn on
      FILE
      PIPE
      PROC
    and turn off all the other selections.
    Click OK to confirm the settings.

  - Close pmos2trc and leave the VIO session open.

== Prepare to start httpd ==

  - Change to your apache2 bin directory.

  - Edit BEGINLIBPATH so that apache2 and php can find the
    modified libc DLLs.

  - Edit BEGINLIBPATH (if needed) so that php.exe can find php5.dll.

  - On my setup this results in

    BEGINLIBPATH=d:\internet\0;D:\Internet\php5;d:\devtools\os2trace;

  - Run

      set LIBPATHSTRICT=T

    to ensure that the modified libc DLLs are used.


-- 
This email was Anti Virus checked by Astaro Security Gateway. http://www.astaro.com

LewisR

2014-12-20 09:22

developer   ~0002943

Steve:

I think I found our problem:

Take a look at PROC002E.TRC and find the entries for PID/TID 002E 0006
which is the thread that's talking to the fung php PID 0033.

If you look at line 2031655, you will find the log message:

002E 0006 | Dos32Write Entry, Return Address = 0x1DA6EAD3 (HTTPD
0001:0000EAD3)
          | Parameter 1: HFILE = 0x00000002
          | Parameter 2: PVOID = 0x02A759EC
          | 0x02A759EC: 5B 53 75 6E 20 4E 6F 76 20 30 39 20 30 31 3A 31
[Sun Nov 09 01:1
          | 0x02A759FC: 39 3A 35 32 20 32 30 31 34 5D 20 5B 65 72 72 6F
9:52 2014] [erro
          | 0x02A75A0C: 72 5D 20 5B 63 6C 69 65 6E 74 20 31 39 32 2E 31
r] [client 192.1
          | 0x02A75A1C: 36 38 2E 31 30 30 2E 38 5D 20 53 63 72 69 70 74
68.100.8] Script
          | 0x02A75A2C: 20 74 69 6D 65 64 20 6F 75 74 20 62 65 66 6F 72
timed out befor
          | 0x02A75A3C: 65 20 72 65 74 75 72 6E 69 6E 67 20 68 65 61 64
e returning head
          | 0x02A75A4C: 65 72 73 3A 20 70 68 70 2E 65 78 65 0D 0A
ers: php.exe..
          | Parameter 3: ULONG = 0x0000006E
          | Parameter 4: PULONG = 0x02A75934

Me thinks the timeouts are too low.

FWIW, this message has been in the logs all along, but probably was easy
to miss among the thousands of other messages.

The failing code is probably near mod_ext_filter.c:829:

        rv = apr_file_read(ctx->proc->out,

The message is coming from util_script.c:443

  else if (rv == -1) {
       ap_log_rerror(APLOG_MARK, APLOG_ERR|APLOG_TOCLIENT, 0, r,
                     "Script timed out before returning headers: %s",
                      apr_filepath_name_get(r->filename));

LewisR

2014-12-20 09:36

developer   ~0002944

Last edited: 2014-12-20 09:38

View 2 revisions

Paul:

I'd *guess* it could be the same error condition found in testpipe,
failing when no data is read from a pipe.....

[Ref: https://groups.google.com/forum/#!topic/apache2/nq7h4_kQnYE ]

Steve:

I'm pretty sure it is. The apr code is abusing the pipe semaphore. The
was the semaphore is supposed to be used is as a signal that it's time to
read from the pipe. The apr code is using the semaphore to wait until
retrying the read. The problem is that this assumes a lot about the order
that thing occur.

What you might want to try is

  do the read
  if the read returns no data
    reset the event semaphore
    wait for the semaphore
    retry the read
    if there's data, good
    if the semaphore timed out fail
    if the semaphore wait was interrupted try again

This way a lost event does can cause spurious errors. At worst the read
will be delayed for a bit.

It's not all that well documented what determines if an can get
interrupted and return ERROR_INTERRUPT. I go by the rule of thumb that if
the API can has a wait associated with it and can be safely restarted if
interrupted, it can return ERROR_INTERRUPT. If an API lists
ERROR_INTERRUPT as a possible return, it's a good idea to check for it.

Paul:

To be clear, this is the code I believe is being used (and failing): /* Unbuffered i/o */
        apr_size_t nbytes;

        if (thefile->pipe)
            DosResetEventSem(thefile->pipeSem, &rc);

        rc = DosRead(thefile->filedes, buf, *len, &nbytes);

        if (rc == ERROR_NO_DATA && thefile->timeout != 0) {
            int rcwait = DosWaitEventSem(thefile->pipeSem, thefile->timeout >= 0 ? thefile->timeout / 1000 : SEM_INDEFINITE_WAIT);

            if (rcwait == 0) {
                rc = DosRead(thefile->filedes, buf, *len, &nbytes);
            }
            else if (rcwait == ERROR_TIMEOUT) {
                *len = 0;
                return APR_TIMEUP;
            }
        }

I did try removing:
        if (thefile->pipe)
            DosResetEventSem(thefile->pipeSem, &rc);

and it made no difference.

Steve:

It's my current guess that the timeouts [during siege tests and referenced in the above Apache/2 list thread with testall] both occurred for the same reason.
A timing race in the apr_file_read code that's supposed to deal with
non-blocking pipes.

[...]

Take a look at the attached [readwrite2.c.diff]. My change is near line 92 of the diff. The idea is that you read until the other side closes the pipe or somethingbad happens and there's a timeout.

LewisR

2014-12-20 09:36

developer  

readwrite.c.diff (11,703 bytes)
Index: readwrite.c
===================================================================
--- readwrite.c	(revision 1629784)
+++ readwrite.c	(working copy)
@@ -24,20 +24,35 @@
 
 #include <malloc.h>
 
-APR_DECLARE(apr_status_t) apr_file_read(apr_file_t *thefile, void *buf, apr_size_t *nbytes)
+APR_DECLARE(apr_status_t) apr_file_read(apr_file_t *thefile, void *buf, apr_size_t *len)
 {
     ULONG rc = 0;
-    ULONG bytesread;
+    ULONG bytes_read = 0;
 
-    if (!thefile->isopen) {
-        *nbytes = 0;
-        return APR_EBADF;
+    if (*len <= 0) {
+        *len = 0;
+        return APR_SUCCESS;
     }
 
+#if 1
+    /* Handle the ungetchar if there is one */
+    /* copied from win32/readwrite.c */
+    if (thefile->ungetchar != -1) {
+        bytes_read = 1;
+        *(char *)buf = (char)thefile->ungetchar;
+        buf = (char *)buf + 1;
+        (*len)--;
+        thefile->ungetchar = -1;
+        if (*len == 0) {
+            *len = bytes_read;
+            return APR_SUCCESS;
+        }
+    }
+#endif
     if (thefile->buffered) {
         char *pos = (char *)buf;
         ULONG blocksize;
-        ULONG size = *nbytes;
+        ULONG size = *len;
 
         apr_thread_mutex_lock(thefile->mutex);
 
@@ -56,20 +71,20 @@
 
         while (rc == 0 && size > 0) {
             if (thefile->bufpos >= thefile->dataRead) {
-                ULONG bytesread;
+                ULONG bytes_read;
                 rc = DosRead(thefile->filedes, thefile->buffer,
-                             thefile->bufsize, &bytesread);
-
-                if (bytesread == 0) {
+                             thefile->bufsize, &bytes_read);
+                if (bytes_read == 0) {
                     if (rc == 0)
                         thefile->eof_hit = TRUE;
                     break;
                 }
-
-                thefile->dataRead = bytesread;
+		else {
+	                thefile->dataRead = bytes_read;
                 thefile->filePtr += thefile->dataRead;
                 thefile->bufpos = 0;
             }
+            }
 
             blocksize = size > thefile->dataRead - thefile->bufpos ? thefile->dataRead - thefile->bufpos : size;
             memcpy(pos, thefile->buffer + thefile->bufpos, blocksize);
@@ -78,40 +93,60 @@
             size -= blocksize;
         }
 
-        *nbytes = rc == 0 ? pos - (char *)buf : 0;
+        *len = rc == 0 ? pos - (char *)buf : 0;
         apr_thread_mutex_unlock(thefile->mutex);
 
-        if (*nbytes == 0 && rc == 0 && thefile->eof_hit) {
+        if (*len == 0 && rc == 0 && thefile->eof_hit) {
             return APR_EOF;
         }
 
         return APR_FROM_OS_ERROR(rc);
     } else {
+        /* Unbuffered i/o */
+        apr_size_t nbytes;
+
+#if 1 // 2014-11-11 SHL
+	for (;;) {
+	    int post_count;
+	    rc = DosRead(thefile->filedes, buf, *len, &nbytes);
+	    if (!thefile->pipe || rc != ERROR_NO_DATA || thefile->timeout == 0)
+	        break;
+	    rc = DosWaitEventSem(thefile->pipeSem, thefile->timeout >= 0 ? thefile->timeout / 1000 : SEM_INDEFINITE_WAIT);
+            DosResetEventSem(thefile->pipeSem, &post_count);
+            if (rc == ERROR_TIMEOUT) {
+                *len = 0;
+                return APR_TIMEUP;
+	    }
+	} // for
+#endif
+
+#if 0 // Old way
         if (thefile->pipe)
             DosResetEventSem(thefile->pipeSem, &rc);
 
-        rc = DosRead(thefile->filedes, buf, *nbytes, &bytesread);
+        rc = DosRead(thefile->filedes, buf, *len, &nbytes);
 
         if (rc == ERROR_NO_DATA && thefile->timeout != 0) {
             int rcwait = DosWaitEventSem(thefile->pipeSem, thefile->timeout >= 0 ? thefile->timeout / 1000 : SEM_INDEFINITE_WAIT);
 
             if (rcwait == 0) {
-                rc = DosRead(thefile->filedes, buf, *nbytes, &bytesread);
+                rc = DosRead(thefile->filedes, buf, *len, &nbytes);
             }
             else if (rcwait == ERROR_TIMEOUT) {
-                *nbytes = 0;
+                *len = 0;
                 return APR_TIMEUP;
             }
         }
+#endif
 
         if (rc) {
-            *nbytes = 0;
+            *len = 0;
             return APR_FROM_OS_ERROR(rc);
         }
 
-        *nbytes = bytesread;
+        *len = nbytes;
         
-        if (bytesread == 0) {
+        if (nbytes == 0) {
             thefile->eof_hit = TRUE;
             return APR_EOF;
         }
@@ -126,17 +161,14 @@
 {
     ULONG rc = 0;
     ULONG byteswritten;
-
     if (!thefile->isopen) {
         *nbytes = 0;
         return APR_EBADF;
     }
-
     if (thefile->buffered) {
         char *pos = (char *)buf;
         int blocksize;
         int size = *nbytes;
-
         apr_thread_mutex_lock(thefile->mutex);
 
         if ( thefile->direction == 0 ) {
@@ -147,19 +179,18 @@
             thefile->bufpos = thefile->dataRead = 0;
             thefile->direction = 1;
         }
-
         while (rc == 0 && size > 0) {
-            if (thefile->bufpos == thefile->bufsize) /* write buffer is full */
+            if (thefile->bufpos == thefile->bufsize){ /* write buffer is full */
                 /* XXX bug; - rc is double-transformed os->apr below */
                 rc = apr_file_flush(thefile);
-
-            blocksize = size > thefile->bufsize - thefile->bufpos ? thefile->bufsize - thefile->bufpos : size;
+		}
+            blocksize = size > thefile->bufsize - thefile->bufpos ? 
+                               thefile->bufsize - thefile->bufpos : size;
             memcpy(thefile->buffer + thefile->bufpos, pos, blocksize);
             thefile->bufpos += blocksize;
             pos += blocksize;
             size -= blocksize;
         }
-
         apr_thread_mutex_unlock(thefile->mutex);
         return APR_FROM_OS_ERROR(rc);
     } else {
@@ -167,20 +198,16 @@
             FILELOCK all = { 0, 0x7fffffff };
             ULONG newpos;
             rc = DosSetFileLocks(thefile->filedes, NULL, &all, -1, 0);
-
             if (rc == 0) {
                 rc = DosSetFilePtr(thefile->filedes, 0, FILE_END, &newpos);
-
                 if (rc == 0) {
-                    rc = DosWrite(thefile->filedes, buf, *nbytes, &byteswritten);
+                    rc = DosWrite(thefile->filedes, (PVOID)buf, *nbytes, &byteswritten);
                 }
-
                 DosSetFileLocks(thefile->filedes, &all, NULL, -1, 0);
             }
         } else {
             rc = DosWrite(thefile->filedes, buf, *nbytes, &byteswritten);
         }
-
         if (rc) {
             *nbytes = 0;
             return APR_FROM_OS_ERROR(rc);
@@ -193,94 +220,84 @@
 
 
 
-#ifdef HAVE_WRITEV
-
 APR_DECLARE(apr_status_t) apr_file_writev(apr_file_t *thefile, const struct iovec *vec, apr_size_t nvec, apr_size_t *nbytes)
 {
-    int bytes;
+    apr_status_t rv = APR_SUCCESS;
+    apr_size_t i;
+    apr_size_t bwrote = 0;
+    char *buf;
 
-    if (thefile->buffered) {
-        apr_status_t rv = apr_file_flush(thefile);
+    *nbytes = 0;
+    for (i = 0; i < nvec; i++) {
+        buf = vec[i].iov_base;
+        bwrote = vec[i].iov_len;
+        rv = apr_file_write(thefile, buf, &bwrote);
+        *nbytes += bwrote;
         if (rv != APR_SUCCESS) {
-            return rv;
+            break;
         }
     }
-
-    if ((bytes = writev(thefile->filedes, vec, nvec)) < 0) {
-        *nbytes = 0;
-        return errno;
+    return rv;
     }
-    else {
-        *nbytes = bytes;
-        return APR_SUCCESS;
-    }
-}
-#endif
 
 
 
 APR_DECLARE(apr_status_t) apr_file_putc(char ch, apr_file_t *thefile)
 {
-    ULONG rc;
-    ULONG byteswritten;
+    apr_size_t nbytes = 1;
 
-    if (!thefile->isopen) {
-        return APR_EBADF;
+    return apr_file_write(thefile, &ch, &nbytes);
     }
 
-    rc = DosWrite(thefile->filedes, &ch, 1, &byteswritten);
 
-    if (rc) {
-        return APR_FROM_OS_ERROR(rc);
-    }
     
-    return APR_SUCCESS;
-}
-
-
-
 APR_DECLARE(apr_status_t) apr_file_ungetc(char ch, apr_file_t *thefile)
 {
-    apr_off_t offset = -1;
-    return apr_file_seek(thefile, APR_CUR, &offset);
+    thefile->ungetchar = (unsigned char)ch;
+    return APR_SUCCESS; 
 }
 
 
 APR_DECLARE(apr_status_t) apr_file_getc(char *ch, apr_file_t *thefile)
 {
-    ULONG rc;
-    apr_size_t bytesread;
+    apr_size_t nbytes = 1;
 
-    if (!thefile->isopen) {
-        return APR_EBADF;
+    return apr_file_read(thefile, ch, &nbytes);
+
     }
 
-    bytesread = 1;
-    rc = apr_file_read(thefile, ch, &bytesread);
 
-    if (rc) {
-        return rc;
-    }
     
-    if (bytesread == 0) {
-        thefile->eof_hit = TRUE;
-        return APR_EOF;
+APR_DECLARE(apr_status_t) apr_file_puts(const char *str, apr_file_t *thefile)
+{
+    return apr_file_write_full(thefile, str, strlen(str), NULL);
     }
     
-    return APR_SUCCESS;
-}
 
 
-
-APR_DECLARE(apr_status_t) apr_file_puts(const char *str, apr_file_t *thefile)
+apr_status_t apr_file_flush_locked(apr_file_t *thefile)
 {
-    apr_size_t len;
+    apr_status_t rv = APR_SUCCESS;
 
-    len = strlen(str);
-    return apr_file_write(thefile, str, &len); 
+    if (thefile->direction == 1 && thefile->bufpos) {
+        apr_ssize_t written;
+
+        do {
+            written = write(thefile->filedes, thefile->buffer, thefile->bufpos);
+        } while (written == -1 && errno == EINTR);
+        if (written == -1) {
+            rv = errno;
+        } else {
+            thefile->filePtr += written;
+            thefile->bufpos = 0;
 }
+    }
 
+    return rv;
+}
 
+
+
 APR_DECLARE(apr_status_t) apr_file_flush(apr_file_t *thefile)
 {
     if (thefile->buffered) {
@@ -349,10 +366,61 @@
 }
 
 
+/* Pull from unix code to start a sync up */
+#if (defined(__INNOTEK_LIBC__) || defined(__WATCOMC__) )
 
+struct apr_file_printf_data {
+    apr_vformatter_buff_t vbuff;
+    apr_file_t *fptr;
+    char *buf;
+};
+
+static int file_printf_flush(apr_vformatter_buff_t *buff)
+{
+    struct apr_file_printf_data *data = (struct apr_file_printf_data *)buff;
+
+    if (apr_file_write_full(data->fptr, data->buf,
+                            data->vbuff.curpos - data->buf, NULL)) {
+        return -1;
+    }
+
+    data->vbuff.curpos = data->buf;
+    return 0;
+}
+
 APR_DECLARE_NONSTD(int) apr_file_printf(apr_file_t *fptr, 
                                         const char *format, ...)
 {
+    struct apr_file_printf_data data;
+    va_list ap;
+    int count;
+
+    /* don't really need a HUGE_STRING_LEN anymore */
+    data.buf = _lmalloc(HUGE_STRING_LEN);
+    if (data.buf == NULL) {
+        return -1;
+    }
+    data.vbuff.curpos = data.buf;
+    data.vbuff.endpos = data.buf + HUGE_STRING_LEN;
+    data.fptr = fptr;
+    va_start(ap, format);
+    count = apr_vformatter(file_printf_flush,
+                           (apr_vformatter_buff_t *)&data, format, ap);
+    /* apr_vformatter does not call flush for the last bits */
+    if (count >= 0) file_printf_flush((apr_vformatter_buff_t *)&data);
+
+    va_end(ap);
+
+    free(data.buf);
+
+    return count;
+}
+
+#else
+
+APR_DECLARE_NONSTD(int) apr_file_printf(apr_file_t *fptr, 
+                                        const char *format, ...)
+{
     int cc;
     va_list ap;
     char *buf;
@@ -369,9 +437,9 @@
     free(buf);
     return (cc == APR_SUCCESS) ? len : -1;
 }
+#endif
 
 
-
 apr_status_t apr_file_check_read(apr_file_t *fd)
 {
     int rc;
readwrite.c.diff (11,703 bytes)

LewisR

2014-12-20 09:38

developer  

readwrite2.c.diff (11,753 bytes)
Index: readwrite.c
===================================================================
--- readwrite.c	(revision 1629784)
+++ readwrite.c	(working copy)
@@ -24,20 +24,35 @@
 
 #include <malloc.h>
 
-APR_DECLARE(apr_status_t) apr_file_read(apr_file_t *thefile, void *buf, apr_size_t *nbytes)
+APR_DECLARE(apr_status_t) apr_file_read(apr_file_t *thefile, void *buf, apr_size_t *len)
 {
     ULONG rc = 0;
-    ULONG bytesread;
+    ULONG bytes_read = 0;
 
-    if (!thefile->isopen) {
-        *nbytes = 0;
-        return APR_EBADF;
+    if (*len <= 0) {
+        *len = 0;
+        return APR_SUCCESS;
     }
 
+#if 1
+    /* Handle the ungetchar if there is one */
+    /* copied from win32/readwrite.c */
+    if (thefile->ungetchar != -1) {
+        bytes_read = 1;
+        *(char *)buf = (char)thefile->ungetchar;
+        buf = (char *)buf + 1;
+        (*len)--;
+        thefile->ungetchar = -1;
+        if (*len == 0) {
+            *len = bytes_read;
+            return APR_SUCCESS;
+        }
+    }
+#endif
     if (thefile->buffered) {
         char *pos = (char *)buf;
         ULONG blocksize;
-        ULONG size = *nbytes;
+        ULONG size = *len;
 
         apr_thread_mutex_lock(thefile->mutex);
 
@@ -56,20 +71,20 @@
 
         while (rc == 0 && size > 0) {
             if (thefile->bufpos >= thefile->dataRead) {
-                ULONG bytesread;
+                ULONG bytes_read;
                 rc = DosRead(thefile->filedes, thefile->buffer,
-                             thefile->bufsize, &bytesread);
-
-                if (bytesread == 0) {
+                             thefile->bufsize, &bytes_read);
+                if (bytes_read == 0) {
                     if (rc == 0)
                         thefile->eof_hit = TRUE;
                     break;
                 }
-
-                thefile->dataRead = bytesread;
+		else {
+	                thefile->dataRead = bytes_read;
                 thefile->filePtr += thefile->dataRead;
                 thefile->bufpos = 0;
             }
+            }
 
             blocksize = size > thefile->dataRead - thefile->bufpos ? thefile->dataRead - thefile->bufpos : size;
             memcpy(pos, thefile->buffer + thefile->bufpos, blocksize);
@@ -78,40 +93,74 @@
             size -= blocksize;
         }
 
-        *nbytes = rc == 0 ? pos - (char *)buf : 0;
+        *len = rc == 0 ? pos - (char *)buf : 0;
         apr_thread_mutex_unlock(thefile->mutex);
 
-        if (*nbytes == 0 && rc == 0 && thefile->eof_hit) {
+        if (*len == 0 && rc == 0 && thefile->eof_hit) {
             return APR_EOF;
         }
 
         return APR_FROM_OS_ERROR(rc);
     } else {
+        /* Unbuffered i/o */
+        apr_size_t nbytes;
+
+#if 1 // 2014-11-11 SHL
+	for (;;) {
+	    int post_count;
         if (thefile->pipe)
+		DosResetEventSem(thefile->pipeSem, &post_count);
+	    rc = DosRead(thefile->filedes, buf, *len, &nbytes);
+	    if (!thefile->pipe)
+		break;			// Blocking i/o - we are done
+	    if (rc == ERROR_MORE_DATA) {
+		// Buffer is full - caller will read more eventually
+		rc = NO_ERROR;
+		break;
+	    }
+	    if (thefile->timeout == 0)
+		break;
+	    if (rc == NO_ERROR) {
+		if (nbytes != 0)
+		    break;
+	    }
+	    else if (rc != ERROR_NO_DATA)
+		break;
+	    rc = DosWaitEventSem(thefile->pipeSem, thefile->timeout >= 0 ? thefile->timeout / 1000 : SEM_INDEFINITE_WAIT);
+	    if (rc == ERROR_TIMEOUT) {
+		*len = 0;
+		return APR_TIMEUP;
+	    }
+	} // for
+#endif
+
+#if 0 // Old way
+	if (thefile->pipe)
             DosResetEventSem(thefile->pipeSem, &rc);
 
-        rc = DosRead(thefile->filedes, buf, *nbytes, &bytesread);
+	rc = DosRead(thefile->filedes, buf, *len, &nbytes);
 
         if (rc == ERROR_NO_DATA && thefile->timeout != 0) {
             int rcwait = DosWaitEventSem(thefile->pipeSem, thefile->timeout >= 0 ? thefile->timeout / 1000 : SEM_INDEFINITE_WAIT);
 
             if (rcwait == 0) {
-                rc = DosRead(thefile->filedes, buf, *nbytes, &bytesread);
+	        rc = DosRead(thefile->filedes, buf, *len, &nbytes);
             }
             else if (rcwait == ERROR_TIMEOUT) {
-                *nbytes = 0;
+	        *len = 0;
                 return APR_TIMEUP;
             }
         }
+#endif
 
         if (rc) {
-            *nbytes = 0;
+	    *len = 0;
             return APR_FROM_OS_ERROR(rc);
         }
 
-        *nbytes = bytesread;
+	*len = nbytes;
         
-        if (bytesread == 0) {
+	if (nbytes == 0) {
             thefile->eof_hit = TRUE;
             return APR_EOF;
         }
@@ -126,17 +175,14 @@
 {
     ULONG rc = 0;
     ULONG byteswritten;
-
     if (!thefile->isopen) {
         *nbytes = 0;
         return APR_EBADF;
     }
-
     if (thefile->buffered) {
         char *pos = (char *)buf;
         int blocksize;
         int size = *nbytes;
-
         apr_thread_mutex_lock(thefile->mutex);
 
         if ( thefile->direction == 0 ) {
@@ -147,19 +193,18 @@
             thefile->bufpos = thefile->dataRead = 0;
             thefile->direction = 1;
         }
-
         while (rc == 0 && size > 0) {
-            if (thefile->bufpos == thefile->bufsize) /* write buffer is full */
+	    if (thefile->bufpos == thefile->bufsize){ /* write buffer is full */
                 /* XXX bug; - rc is double-transformed os->apr below */
                 rc = apr_file_flush(thefile);
-
-            blocksize = size > thefile->bufsize - thefile->bufpos ? thefile->bufsize - thefile->bufpos : size;
+		}
+	    blocksize = size > thefile->bufsize - thefile->bufpos ? 
+	                       thefile->bufsize - thefile->bufpos : size;
             memcpy(thefile->buffer + thefile->bufpos, pos, blocksize);
             thefile->bufpos += blocksize;
             pos += blocksize;
             size -= blocksize;
         }
-
         apr_thread_mutex_unlock(thefile->mutex);
         return APR_FROM_OS_ERROR(rc);
     } else {
@@ -167,20 +212,16 @@
             FILELOCK all = { 0, 0x7fffffff };
             ULONG newpos;
             rc = DosSetFileLocks(thefile->filedes, NULL, &all, -1, 0);
-
             if (rc == 0) {
                 rc = DosSetFilePtr(thefile->filedes, 0, FILE_END, &newpos);
-
                 if (rc == 0) {
-                    rc = DosWrite(thefile->filedes, buf, *nbytes, &byteswritten);
+	            rc = DosWrite(thefile->filedes, (PVOID)buf, *nbytes, &byteswritten);
                 }
-
                 DosSetFileLocks(thefile->filedes, &all, NULL, -1, 0);
             }
         } else {
             rc = DosWrite(thefile->filedes, buf, *nbytes, &byteswritten);
         }
-
         if (rc) {
             *nbytes = 0;
             return APR_FROM_OS_ERROR(rc);
@@ -193,94 +234,84 @@
 
 
 
-#ifdef HAVE_WRITEV
-
 APR_DECLARE(apr_status_t) apr_file_writev(apr_file_t *thefile, const struct iovec *vec, apr_size_t nvec, apr_size_t *nbytes)
 {
-    int bytes;
+    apr_status_t rv = APR_SUCCESS;
+    apr_size_t i;
+    apr_size_t bwrote = 0;
+    char *buf;
 
-    if (thefile->buffered) {
-        apr_status_t rv = apr_file_flush(thefile);
+    *nbytes = 0;
+    for (i = 0; i < nvec; i++) {
+	buf = vec[i].iov_base;
+	bwrote = vec[i].iov_len;
+	rv = apr_file_write(thefile, buf, &bwrote);
+	*nbytes += bwrote;
         if (rv != APR_SUCCESS) {
-            return rv;
+	    break;
         }
     }
-
-    if ((bytes = writev(thefile->filedes, vec, nvec)) < 0) {
-        *nbytes = 0;
-        return errno;
+    return rv;
     }
-    else {
-        *nbytes = bytes;
-        return APR_SUCCESS;
-    }
-}
-#endif
 
 
 
 APR_DECLARE(apr_status_t) apr_file_putc(char ch, apr_file_t *thefile)
 {
-    ULONG rc;
-    ULONG byteswritten;
+    apr_size_t nbytes = 1;
 
-    if (!thefile->isopen) {
-        return APR_EBADF;
+    return apr_file_write(thefile, &ch, &nbytes);
     }
 
-    rc = DosWrite(thefile->filedes, &ch, 1, &byteswritten);
 
-    if (rc) {
-        return APR_FROM_OS_ERROR(rc);
-    }
     
-    return APR_SUCCESS;
-}
-
-
-
 APR_DECLARE(apr_status_t) apr_file_ungetc(char ch, apr_file_t *thefile)
 {
-    apr_off_t offset = -1;
-    return apr_file_seek(thefile, APR_CUR, &offset);
+    thefile->ungetchar = (unsigned char)ch;
+    return APR_SUCCESS; 
 }
 
 
 APR_DECLARE(apr_status_t) apr_file_getc(char *ch, apr_file_t *thefile)
 {
-    ULONG rc;
-    apr_size_t bytesread;
+    apr_size_t nbytes = 1;
 
-    if (!thefile->isopen) {
-        return APR_EBADF;
+    return apr_file_read(thefile, ch, &nbytes);
+
     }
 
-    bytesread = 1;
-    rc = apr_file_read(thefile, ch, &bytesread);
 
-    if (rc) {
-        return rc;
-    }
     
-    if (bytesread == 0) {
-        thefile->eof_hit = TRUE;
-        return APR_EOF;
+APR_DECLARE(apr_status_t) apr_file_puts(const char *str, apr_file_t *thefile)
+{
+    return apr_file_write_full(thefile, str, strlen(str), NULL);
     }
     
-    return APR_SUCCESS;
-}
 
 
-
-APR_DECLARE(apr_status_t) apr_file_puts(const char *str, apr_file_t *thefile)
+apr_status_t apr_file_flush_locked(apr_file_t *thefile)
 {
-    apr_size_t len;
+    apr_status_t rv = APR_SUCCESS;
 
-    len = strlen(str);
-    return apr_file_write(thefile, str, &len); 
+    if (thefile->direction == 1 && thefile->bufpos) {
+	apr_ssize_t written;
+
+	do {
+	    written = write(thefile->filedes, thefile->buffer, thefile->bufpos);
+	} while (written == -1 && errno == EINTR);
+	if (written == -1) {
+	    rv = errno;
+	} else {
+	    thefile->filePtr += written;
+	    thefile->bufpos = 0;
 }
+    }
 
+    return rv;
+}
 
+
+
 APR_DECLARE(apr_status_t) apr_file_flush(apr_file_t *thefile)
 {
     if (thefile->buffered) {
@@ -349,10 +380,61 @@
 }
 
 
+/* Pull from unix code to start a sync up */
+#if (defined(__INNOTEK_LIBC__) || defined(__WATCOMC__) )
 
+struct apr_file_printf_data {
+    apr_vformatter_buff_t vbuff;
+    apr_file_t *fptr;
+    char *buf;
+};
+
+static int file_printf_flush(apr_vformatter_buff_t *buff)
+{
+    struct apr_file_printf_data *data = (struct apr_file_printf_data *)buff;
+
+    if (apr_file_write_full(data->fptr, data->buf,
+	                    data->vbuff.curpos - data->buf, NULL)) {
+	return -1;
+    }
+
+    data->vbuff.curpos = data->buf;
+    return 0;
+}
+
 APR_DECLARE_NONSTD(int) apr_file_printf(apr_file_t *fptr, 
                                         const char *format, ...)
 {
+    struct apr_file_printf_data data;
+    va_list ap;
+    int count;
+
+    /* don't really need a HUGE_STRING_LEN anymore */
+    data.buf = _lmalloc(HUGE_STRING_LEN);
+    if (data.buf == NULL) {
+	return -1;
+    }
+    data.vbuff.curpos = data.buf;
+    data.vbuff.endpos = data.buf + HUGE_STRING_LEN;
+    data.fptr = fptr;
+    va_start(ap, format);
+    count = apr_vformatter(file_printf_flush,
+	                   (apr_vformatter_buff_t *)&data, format, ap);
+    /* apr_vformatter does not call flush for the last bits */
+    if (count >= 0) file_printf_flush((apr_vformatter_buff_t *)&data);
+
+    va_end(ap);
+
+    free(data.buf);
+
+    return count;
+}
+
+#else
+
+APR_DECLARE_NONSTD(int) apr_file_printf(apr_file_t *fptr, 
+	                                const char *format, ...)
+{
     int cc;
     va_list ap;
     char *buf;
@@ -369,9 +451,9 @@
     free(buf);
     return (cc == APR_SUCCESS) ? len : -1;
 }
+#endif
 
 
-
 apr_status_t apr_file_check_read(apr_file_t *fd)
 {
     int rc;
readwrite2.c.diff (11,753 bytes)

LewisR

2014-12-20 09:42

developer   ~0002945

Steve [upon revising original readwrite.c.diff]:

It occurred to me that we need to map ERROR_MORE_DATA to NO_ERROR because,
like ERROR_NO_DATA, it is a notice, not an error. It means that the
caller's buffer is full and the pipe is not empty. I also added a bit
more code that should ensure that we don't confuse no data currently
available with end of data.

What I'd like to know for sure is if NO_ERROR and count = 0 always means
end of data, even in the non-blocking case. This is possible if the
kernel ensures ERROR_NO_DATA is always returned if an open, non-blocking
pipe has no data available. The docs are not 100% clear if this is the
case.

Could you instrument testpipe and readchild to trace as described in
"testing-http-php-with-os2trace.txt" and see what happens. You don't need
to instrument libc066 because apr is using native DOS APIs. Since we care
about the semaphore activity, add SEM to the selected DOSCALLS API groups.

Paul:

Unfortunately no change.

Steve:

Not on the gross pass/fail level, but the .trc files show a lot. I think
we are close to resolving the httpd issues. The testall failure is
something else.

Here's what's happening.

Testall fails with

0CC1 0001 | Dos32Close Entry, Return Address = 0x00040459 (TESTALL
0001:00030459)
          | Parameter 1: HFILE = 0x00000006

0CC1 0001 | Dos32Close Exit
     PASS | Return code: 0 (NO_ERROR)

0CC1 0001 | Dos32ResetEventSem Entry, Return Address = 0x00040B03 (TESTALL
0001:00030B03)
          | Parameter 1: HEV = 0x80010187
          | Parameter 2: PULONG = 0x001CFDAC

0CC1 0001 | Dos32ResetEventSem Exit
     FAIL | Return code: 300 (ERROR_ALREADY_RESET)
          | Parameter 2: PULONG = 0x001CFDAC [0x00000000]

0CC1 0001 | Dos32Read Entry, Return Address = 0x00040B28 (TESTALL
0001:00030B28)
          | Parameter 1: HFILE = 0x00000007
          | Parameter 2: PVOID = 0x001CFE28
          | Parameter 3: ULONG = 0x00000080
          | Parameter 4: PULONG = 0x001CFDB0

0CC1 0001 | Dos32Read Exit
     FAIL | Return code: 232 (ERROR_NO_DATA)
          | Parameter 2: PVOID = 0x001CFE28
          | Parameter 4: PULONG = 0x001CFDB0 [0x00000000]

0CC1 0001 | Dos32WaitEventSem Entry, Return Address = 0x00040BB3 (TESTALL
0001:00030BB3)
          | Parameter 1: HEV = 0x80010187
          | Parameter 2: ULONG = 0x00002710

0CC1 0001 | Dos32WaitEventSem Exit
     FAIL | Return code: 640 (ERROR_TIMEOUT)
          | API entry at 09:58:50.18, exit at 09:59:00.20

The code is doing what is supposed to - wait 10 seconds for semaphore to
be posted when the read reports no data.

The problem is on the other side, the readchild trace reports

0CC2 0001 | Dos32Read Entry, Return Address = 0x00012928 (READCHIL
0001:00002928)
          | Parameter 1: HFILE = 0x00000000
          | Parameter 2: PVOID = 0x0013FEF8
          | Parameter 3: ULONG = 0x00000080
          | Parameter 4: PULONG = 0x0013FEB0

0CC2 0001 | Dos32Read Exit
     PASS | Return code: 0 (NO_ERROR)
          | Parameter 2: PVOID = 0x0013FEF8
          | 0x0013FEF8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
 ................
          | 0x0013FF08: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
 ................
          | 0x0013FF18: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
 ................
          | 0x0013FF28: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
 ................
          | 0x0013FF38: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
 ................
          | 0x0013FF48: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
 ................
          | 0x0013FF58: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
 ................
          | 0x0013FF68: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
 ................
          | Parameter 4: PULONG = 0x0013FEB0 [0x00000080]

0CC2 0001 | Dos32Read Entry, Return Address = 0x00012928 (READCHIL
0001:00002928)
          | Parameter 1: HFILE = 0x00000000
          | Parameter 2: PVOID = 0x0013FEF8
          | Parameter 3: ULONG = 0x00000080
          | Parameter 4: PULONG = 0x0013FEB0

READCHILD.EXE (0CC2) stopping at 09:59:05.07 on 11/12/2014

Readchild is still busy reading data when testall decides it has waited
long enough for readchild to respond. It's almost as if readchild does
not get informed that the testpipe has closed the other end of the pipe.

I need to think about what, if anything, that the APR might be doing to
cause this.

Note that on the readchild side stdin looks like a file (i.e. the pipe
flag is not set) so reads are normal blocking reads.

LewisR

2014-12-20 09:46

developer   ~0002946

Paul:

Given your comments I'll rebuild httpd.dll anyway and let Lewis run the
siege tests.

Steve:

Good idea. If it works, I will tune my patch to avoid what I believe is
an extra sem wait, which will improve performance. I found some
additional references to ERROR_NO_DATA in the CPG&R docs that strenghten
my belief that the statement found elsewhere that states

<snip>
  If you attempt a read of a named pipe when the other end of the pipe is
  closed while the read is pending, DosRead returns:

       pcbActual equal to 0
       ulrc equal to either 0 or ERROR_NO_DATA
</snip>

is probably not correct. We sould be able to depend on NO_ERROR with
actual count = 0 meaning that the other side has closed the pipe.

[Paul built test build of httpd.dll.]

Steve:

Assuming nothing really bad shows up with your last build, here's the next
patch that I would like to test [readwrite3.c.diff]. It assumes that NO_ERROR and count = 0 indicates end of data and bypasses the semaphore wait.

LewisR

2014-12-20 09:46

developer  

readwrite3.c.diff (11,964 bytes)
Index: readwrite.c
===================================================================
--- readwrite.c	(revision 1629784)
+++ readwrite.c	(working copy)
@@ -24,20 +24,34 @@
 
 #include <malloc.h>
 
-APR_DECLARE(apr_status_t) apr_file_read(apr_file_t *thefile, void *buf, apr_size_t *nbytes)
+APR_DECLARE(apr_status_t) apr_file_read(apr_file_t *thefile, void *buf, apr_size_t *len)
 {
     ULONG rc = 0;
-    ULONG bytesread;
+    ULONG bytes_read = 0;
 
-    if (!thefile->isopen) {
-        *nbytes = 0;
-        return APR_EBADF;
+    if (*len <= 0) {
+        *len = 0;
+        return APR_SUCCESS;
     }
-
+#if 1
+    /* Handle the ungetchar if there is one */
+    /* copied from win32/readwrite.c */
+    if (thefile->ungetchar != -1) {
+        bytes_read = 1;
+        *(char *)buf = (char)thefile->ungetchar;
+        buf = (char *)buf + 1;
+        (*len)--;
+        thefile->ungetchar = -1;
+        if (*len == 0) {
+            *len = bytes_read;
+            return APR_SUCCESS;
+        }
+    }
+#endif
     if (thefile->buffered) {
         char *pos = (char *)buf;
         ULONG blocksize;
-        ULONG size = *nbytes;
+        ULONG size = *len;
 
         apr_thread_mutex_lock(thefile->mutex);
 
@@ -56,20 +70,20 @@
 
         while (rc == 0 && size > 0) {
             if (thefile->bufpos >= thefile->dataRead) {
-                ULONG bytesread;
+                ULONG bytes_read;
                 rc = DosRead(thefile->filedes, thefile->buffer,
-                             thefile->bufsize, &bytesread);
-
-                if (bytesread == 0) {
+                             thefile->bufsize, &bytes_read);
+                if (bytes_read == 0) {
                     if (rc == 0)
                         thefile->eof_hit = TRUE;
                     break;
                 }
-
-                thefile->dataRead = bytesread;
+		else {
+	                thefile->dataRead = bytes_read;
                 thefile->filePtr += thefile->dataRead;
                 thefile->bufpos = 0;
             }
+            }
 
             blocksize = size > thefile->dataRead - thefile->bufpos ? thefile->dataRead - thefile->bufpos : size;
             memcpy(pos, thefile->buffer + thefile->bufpos, blocksize);
@@ -78,40 +92,80 @@
             size -= blocksize;
         }
 
-        *nbytes = rc == 0 ? pos - (char *)buf : 0;
+        *len = rc == 0 ? pos - (char *)buf : 0;
         apr_thread_mutex_unlock(thefile->mutex);
 
-        if (*nbytes == 0 && rc == 0 && thefile->eof_hit) {
+        if (*len == 0 && rc == 0 && thefile->eof_hit) {
             return APR_EOF;
         }
 
         return APR_FROM_OS_ERROR(rc);
     } else {
+        /* Unbuffered i/o */
+        apr_size_t nbytes;
+
+#if 1 // 2014-11-11 SHL
+	for (;;) {
+	    int post_count;
         if (thefile->pipe)
+		DosResetEventSem(thefile->pipeSem, &post_count);
+	    rc = DosRead(thefile->filedes, buf, *len, &nbytes);
+	    if (!thefile->pipe)
+		break;			// Blocking i/o - we are done
+	    if (rc == ERROR_MORE_DATA) {
+		// Buffer is full - caller will read more eventually
+		rc = NO_ERROR;
+		break;
+	    }
+	    if (thefile->timeout == 0)
+		break;
+#if 0 // 2014-11-12 SHL	assumes eos can return NO_ERROR or ERROR_NO_DATA
+	    if (rc == NO_ERROR) {
+		if (nbytes != 0)
+		    break;
+	    }
+	    else if (rc != ERROR_NO_DATA)
+		break;
+#endif
+#if 1 // 2014-11-12 SHL	assumes eos can only return NO_ERROR and count 0
+	    if (rc != ERROR_NO_DATA)
+		break;
+#endif
+	    rc = DosWaitEventSem(thefile->pipeSem, thefile->timeout >= 0 ? thefile->timeout / 1000 : SEM_INDEFINITE_WAIT);
+	    if (rc == ERROR_TIMEOUT) {
+		*len = 0;
+		return APR_TIMEUP;
+	    }
+	} // for
+#endif
+
+#if 0 // Old way
+	if (thefile->pipe)
             DosResetEventSem(thefile->pipeSem, &rc);
 
-        rc = DosRead(thefile->filedes, buf, *nbytes, &bytesread);
+	rc = DosRead(thefile->filedes, buf, *len, &nbytes);
 
         if (rc == ERROR_NO_DATA && thefile->timeout != 0) {
             int rcwait = DosWaitEventSem(thefile->pipeSem, thefile->timeout >= 0 ? thefile->timeout / 1000 : SEM_INDEFINITE_WAIT);
 
             if (rcwait == 0) {
-                rc = DosRead(thefile->filedes, buf, *nbytes, &bytesread);
+	        rc = DosRead(thefile->filedes, buf, *len, &nbytes);
             }
             else if (rcwait == ERROR_TIMEOUT) {
-                *nbytes = 0;
+	        *len = 0;
                 return APR_TIMEUP;
             }
         }
+#endif
 
         if (rc) {
-            *nbytes = 0;
+	    *len = 0;
             return APR_FROM_OS_ERROR(rc);
         }
 
-        *nbytes = bytesread;
+	*len = nbytes;
         
-        if (bytesread == 0) {
+	if (nbytes == 0) {
             thefile->eof_hit = TRUE;
             return APR_EOF;
         }
@@ -126,17 +180,14 @@
 {
     ULONG rc = 0;
     ULONG byteswritten;
-
     if (!thefile->isopen) {
         *nbytes = 0;
         return APR_EBADF;
     }
-
     if (thefile->buffered) {
         char *pos = (char *)buf;
         int blocksize;
         int size = *nbytes;
-
         apr_thread_mutex_lock(thefile->mutex);
 
         if ( thefile->direction == 0 ) {
@@ -147,19 +198,18 @@
             thefile->bufpos = thefile->dataRead = 0;
             thefile->direction = 1;
         }
-
         while (rc == 0 && size > 0) {
-            if (thefile->bufpos == thefile->bufsize) /* write buffer is full */
+	    if (thefile->bufpos == thefile->bufsize){ /* write buffer is full */
                 /* XXX bug; - rc is double-transformed os->apr below */
                 rc = apr_file_flush(thefile);
-
-            blocksize = size > thefile->bufsize - thefile->bufpos ? thefile->bufsize - thefile->bufpos : size;
+		}
+	    blocksize = size > thefile->bufsize - thefile->bufpos ? 
+	                       thefile->bufsize - thefile->bufpos : size;
             memcpy(thefile->buffer + thefile->bufpos, pos, blocksize);
             thefile->bufpos += blocksize;
             pos += blocksize;
             size -= blocksize;
         }
-
         apr_thread_mutex_unlock(thefile->mutex);
         return APR_FROM_OS_ERROR(rc);
     } else {
@@ -167,20 +217,16 @@
             FILELOCK all = { 0, 0x7fffffff };
             ULONG newpos;
             rc = DosSetFileLocks(thefile->filedes, NULL, &all, -1, 0);
-
             if (rc == 0) {
                 rc = DosSetFilePtr(thefile->filedes, 0, FILE_END, &newpos);
-
                 if (rc == 0) {
-                    rc = DosWrite(thefile->filedes, buf, *nbytes, &byteswritten);
+	            rc = DosWrite(thefile->filedes, (PVOID)buf, *nbytes, &byteswritten);
                 }
-
                 DosSetFileLocks(thefile->filedes, &all, NULL, -1, 0);
             }
         } else {
             rc = DosWrite(thefile->filedes, buf, *nbytes, &byteswritten);
         }
-
         if (rc) {
             *nbytes = 0;
             return APR_FROM_OS_ERROR(rc);
@@ -193,94 +239,84 @@
 
 
 
-#ifdef HAVE_WRITEV
-
 APR_DECLARE(apr_status_t) apr_file_writev(apr_file_t *thefile, const struct iovec *vec, apr_size_t nvec, apr_size_t *nbytes)
 {
-    int bytes;
+    apr_status_t rv = APR_SUCCESS;
+    apr_size_t i;
+    apr_size_t bwrote = 0;
+    char *buf;
 
-    if (thefile->buffered) {
-        apr_status_t rv = apr_file_flush(thefile);
+    *nbytes = 0;
+    for (i = 0; i < nvec; i++) {
+	buf = vec[i].iov_base;
+	bwrote = vec[i].iov_len;
+	rv = apr_file_write(thefile, buf, &bwrote);
+	*nbytes += bwrote;
         if (rv != APR_SUCCESS) {
-            return rv;
+	    break;
         }
     }
-
-    if ((bytes = writev(thefile->filedes, vec, nvec)) < 0) {
-        *nbytes = 0;
-        return errno;
+    return rv;
     }
-    else {
-        *nbytes = bytes;
-        return APR_SUCCESS;
-    }
-}
-#endif
 
 
 
 APR_DECLARE(apr_status_t) apr_file_putc(char ch, apr_file_t *thefile)
 {
-    ULONG rc;
-    ULONG byteswritten;
+    apr_size_t nbytes = 1;
 
-    if (!thefile->isopen) {
-        return APR_EBADF;
+    return apr_file_write(thefile, &ch, &nbytes);
     }
 
-    rc = DosWrite(thefile->filedes, &ch, 1, &byteswritten);
 
-    if (rc) {
-        return APR_FROM_OS_ERROR(rc);
-    }
     
-    return APR_SUCCESS;
-}
-
-
-
 APR_DECLARE(apr_status_t) apr_file_ungetc(char ch, apr_file_t *thefile)
 {
-    apr_off_t offset = -1;
-    return apr_file_seek(thefile, APR_CUR, &offset);
+    thefile->ungetchar = (unsigned char)ch;
+    return APR_SUCCESS; 
 }
 
 
 APR_DECLARE(apr_status_t) apr_file_getc(char *ch, apr_file_t *thefile)
 {
-    ULONG rc;
-    apr_size_t bytesread;
+    apr_size_t nbytes = 1;
 
-    if (!thefile->isopen) {
-        return APR_EBADF;
+    return apr_file_read(thefile, ch, &nbytes);
+
     }
 
-    bytesread = 1;
-    rc = apr_file_read(thefile, ch, &bytesread);
 
-    if (rc) {
-        return rc;
-    }
     
-    if (bytesread == 0) {
-        thefile->eof_hit = TRUE;
-        return APR_EOF;
+APR_DECLARE(apr_status_t) apr_file_puts(const char *str, apr_file_t *thefile)
+{
+    return apr_file_write_full(thefile, str, strlen(str), NULL);
     }
     
-    return APR_SUCCESS;
-}
 
 
-
-APR_DECLARE(apr_status_t) apr_file_puts(const char *str, apr_file_t *thefile)
+apr_status_t apr_file_flush_locked(apr_file_t *thefile)
 {
-    apr_size_t len;
+    apr_status_t rv = APR_SUCCESS;
 
-    len = strlen(str);
-    return apr_file_write(thefile, str, &len); 
+    if (thefile->direction == 1 && thefile->bufpos) {
+	apr_ssize_t written;
+
+	do {
+	    written = write(thefile->filedes, thefile->buffer, thefile->bufpos);
+	} while (written == -1 && errno == EINTR);
+	if (written == -1) {
+	    rv = errno;
+	} else {
+	    thefile->filePtr += written;
+	    thefile->bufpos = 0;
 }
+    }
 
+    return rv;
+}
 
+
+
 APR_DECLARE(apr_status_t) apr_file_flush(apr_file_t *thefile)
 {
     if (thefile->buffered) {
@@ -349,10 +385,61 @@
 }
 
 
+/* Pull from unix code to start a sync up */
+#if (defined(__INNOTEK_LIBC__) || defined(__WATCOMC__) )
 
+struct apr_file_printf_data {
+    apr_vformatter_buff_t vbuff;
+    apr_file_t *fptr;
+    char *buf;
+};
+
+static int file_printf_flush(apr_vformatter_buff_t *buff)
+{
+    struct apr_file_printf_data *data = (struct apr_file_printf_data *)buff;
+
+    if (apr_file_write_full(data->fptr, data->buf,
+	                    data->vbuff.curpos - data->buf, NULL)) {
+	return -1;
+    }
+
+    data->vbuff.curpos = data->buf;
+    return 0;
+}
+
 APR_DECLARE_NONSTD(int) apr_file_printf(apr_file_t *fptr, 
                                         const char *format, ...)
 {
+    struct apr_file_printf_data data;
+    va_list ap;
+    int count;
+
+    /* don't really need a HUGE_STRING_LEN anymore */
+    data.buf = _lmalloc(HUGE_STRING_LEN);
+    if (data.buf == NULL) {
+	return -1;
+    }
+    data.vbuff.curpos = data.buf;
+    data.vbuff.endpos = data.buf + HUGE_STRING_LEN;
+    data.fptr = fptr;
+    va_start(ap, format);
+    count = apr_vformatter(file_printf_flush,
+	                   (apr_vformatter_buff_t *)&data, format, ap);
+    /* apr_vformatter does not call flush for the last bits */
+    if (count >= 0) file_printf_flush((apr_vformatter_buff_t *)&data);
+
+    va_end(ap);
+
+    free(data.buf);
+
+    return count;
+}
+
+#else
+
+APR_DECLARE_NONSTD(int) apr_file_printf(apr_file_t *fptr, 
+	                                const char *format, ...)
+{
     int cc;
     va_list ap;
     char *buf;
@@ -369,9 +456,9 @@
     free(buf);
     return (cc == APR_SUCCESS) ? len : -1;
 }
+#endif
 
 
-
 apr_status_t apr_file_check_read(apr_file_t *fd)
 {
     int rc;
readwrite3.c.diff (11,964 bytes)

LewisR

2014-12-20 09:53

developer   ~0002947

Lewis [upon testing Paul's build before readwrite3.c.diff]:

Fewer stuck ones [PHP processes] than before, but I have not looked to see how these compare to the last batch.

Steve:

Better, but not fixed. My testing indicates that the last patch I posted
is needed to ensure that end of data is seen on the httpd side.

Paul:

Build with Steven's latest patch:
http://smedley.id.au/tmp/httpd-dll-20141114.zip

Steve:

Ignoring Lewis's issues, my traces look exactly like I expect except for
one thing.

This from the httpd .trc:

0C27 0003 | Dos32Read Entry at 19:41:05.88, Return Address = 0x0F4AE788
(HTTPD 0001:0000E788)
          | Parameter 1: HFILE = 0x0000000E
          | Parameter 2: PVOID = 0x006DAD68
          | Parameter 3: ULONG = 0x00001F40
          | Parameter 4: PULONG = 0x02829A6C

0C27 0003 | Dos32Read Exit at 19:41:05.88
     PASS | Return code: 0 (NO_ERROR)
          | Parameter 2: PVOID = 0x006DAD68
          | 0x006DAD68: 58 2D 50 6F 77 65 72 65 64 2D 42 79 3A 20 50 48
X-Powered-By: PH
          | 0x006DAD78: 50 2F 35 2E 34 2E 33 34 0D 0A 43 6F 6E 74 65 6E
P/5.4.34..Conten
          | 0x006DAD88: 74 2D 74 79 70 65 3A 20 74 65 78 74 2F 68 74 6D
t-type: text/htm
          | 0x006DAD98: 6C 0D 0A 0D 0A 3C 68 74 6D 6C 3E 0D 0A 20 3C 68
l....<html>.. <h
          | 0x006DADA8: 65 61 64 3E 0D 0A 20 20 3C 74 69 74 6C 65 3E 50
ead>.. <title>P
          | 0x006DADB8: 48 50 20 54 65 73 74 20 2D 20 6F 73 32 74 72 61
HP Test - os2tra
          | 0x006DADC8: 63 65 3C 2F 74 69 74 6C 65 3E 0D 0A 20 3C 2F 68
ce</title>.. </h
          | 0x006DADD8: 65 61 64 3E 0D 0A 20 3C 62 6F 64 79 3E 0D 0A 20
ead>.. <body>..
          | 0x006DADE8: 20 20 48 65 6C 6C 6F 20 66 72 6F 6D 20 4D 72 2E
Hello from Mr.
          | 0x006DADF8: 20 68 65 6C 6C 6F 2E 70 68 70 27 73 20 68 74 6D
hello.php's htm
          | 0x006DAE08: 6C 20 63 6F 64 65 2E 0D 0A 20 20 20 3C 70 3E 48
l code...

H
          | 0x006DAE18: 65 6C 6C 6F 20 66 72 6F 6D 20 4D 72 2E 20 68 65
ello from Mr. he
          | 0x006DAE28: 6C 6C 6F 2E 70 68 70 27 73 20 50 48 50 20 63 6F
llo.php's PHP co
          | 0x006DAE38: 64 65 2E 0A 3C 2F 70 3E 20 3C 2F 62 6F 64 79 3E
de..

</body>
          | 0x006DAE48: 0D 0A 3C 2F 68 74 6D 6C 3E 0D 0A
..</html>..
          | Parameter 4: PULONG = 0x02829A6C [0x000000EB]

At this point the read apr_file_read() returns to the caller and the
caller calls apr_file_read() for more data...

0C27 0003 | Dos32ResetEventSem Entry at 19:41:05.88, Return Address =
0x0F4AE763 (HTTPD 0001:0000E763)
          | Parameter 1: HEV = 0x8001019A
          | Parameter 2: PULONG = 0x0282BA28

0C27 0003 | Dos32ResetEventSem Exit at 19:41:05.88
     FAIL | Return code: 300 (ERROR_ALREADY_RESET)
          | Parameter 2: PULONG = 0x0282BA28 [0x00000000]

0C27 0003 | Dos32Read Entry at 19:41:05.88, Return Address = 0x0F4AE788
(HTTPD 0001:0000E788)
          | Parameter 1: HFILE = 0x0000000E
          | Parameter 2: PVOID = 0x006DEE28
          | Parameter 3: ULONG = 0x00001F40
          | Parameter 4: PULONG = 0x0282BA2C

0C27 0003 | Dos32Read Exit at 19:41:05.88
     FAIL | Return code: 232 (ERROR_NO_DATA)
          | Parameter 2: PVOID = 0x006DEE28
          | Parameter 4: PULONG = 0x0282BA2C [0x00000000]

This is OK because php.exe has no more data to send and has not yet shut
itself down.

However, this is really wierd:

0C27 0003 | Dos32RequestMutexSem Entry at 19:41:05.88, Return Address =
0x0F4C7CA9 (HTTPD 0001:00027CA9)
          | Parameter 1: HMTX = 0x0001006C
          | Parameter 2: ULONG = 0xFFFFFFFF

0C27 0003 | Dos32RequestMutexSem Exit at 19:41:05.88
     PASS | Return code: 0 (NO_ERROR)
          | API entry at 19:41:05.88, exit at 19:41:05.88

0C27 0003 | Dos32ReleaseMutexSem Entry at 19:41:05.88, Return Address =
0x0F4C7D0D (HTTPD 0001:00027D0D)
          | Parameter 1: HMTX = 0x0001006C

0C27 0003 | Dos32ReleaseMutexSem Exit at 19:41:05.88
     PASS | Return code: 0 (NO_ERROR)

My sources contain no mutex request. I don't have a current map, but I
suspect the call is from apr_thread_mutex_lock()

0C27 0003 | Dos32RequestMutexSem Entry at 19:41:05.88, Return Address =
0x0F4C7CA9 (HTTPD 0001:00027CA9)
          | Parameter 1: HMTX = 0x0001006C
          | Parameter 2: ULONG = 0xFFFFFFFF

0C27 0003 | Dos32RequestMutexSem Exit at 19:41:05.88
     PASS | Return code: 0 (NO_ERROR)
          | API entry at 19:41:05.88, exit at 19:41:05.88

0C27 0003 | Dos32ReleaseMutexSem Entry at 19:41:05.88, Return Address =
0x0F4C7D0D (HTTPD 0001:00027D0D)
          | Parameter 1: HMTX = 0x0001006C

0C27 0003 | Dos32ReleaseMutexSem Exit at 19:41:05.88
     PASS | Return code: 0 (NO_ERROR)

Here we loop back to the top a try the read again...

0C27 0003 | Dos32ResetEventSem Entry at 19:41:05.93, Return Address =
0x0F4AE763 (HTTPD 0001:0000E763)
          | Parameter 1: HEV = 0x8001019A
          | Parameter 2: PULONG = 0x0282BA28

0C27 0003 | Dos32ResetEventSem Exit at 19:41:05.93
     FAIL | Return code: 300 (ERROR_ALREADY_RESET)
          | Parameter 2: PULONG = 0x0282BA28 [0x00000000]

0C27 0003 | Dos32Read Entry at 19:41:05.93, Return Address = 0x0F4AE788
(HTTPD 0001:0000E788)
          | Parameter 1: HFILE = 0x0000000E
          | Parameter 2: PVOID = 0x006DAD68
          | Parameter 3: ULONG = 0x00001F40
          | Parameter 4: PULONG = 0x0282BA2C

0C27 0003 | Dos32Read Exit at 19:41:05.93
     FAIL | Return code: 232 (ERROR_NO_DATA)
          | Parameter 2: PVOID = 0x006DAD68
          | Parameter 4: PULONG = 0x0282BA2C [0x00000000]

This loop continues on until the read returns NO_ERROR and count 0

Paul, do you have any idea where the mutex calls are coming from?

Lewis:

Looks like we're moving in the right direction. Fewer stuck processes,
and the PHP instances *were* killable if I *first* used the normal kill
from top and then the force kill option. httpd instances did not want to
shutdown from -k, but they did kill, all but one last child, which I had
to take out with Ctrl-C.

Steve:

That is better.

You still seem to have the stuck write issue. The problem seems to be on
the httpd end because I can see the timed out messages going to the log.
It's hard to believe the php code is this slow, but maybe it is

Let's try this. Add SEM to the traced items and Use
Customize->Timestamping and enable timestamping. This will allow us the
see when php writes relative to when httpd times out.

Paul [responding to Steve's earlier question about the origin of the mutex calls]:

grepping apache2 and apr source, DosRequestMutexSem could be from:
Apache2:
server\mpm\mpmt_os2\mpmt_os2_child.c

APR:
locks\os2\proc_mutex.c
locks\os2\thread_mutex.c
locks\os2\thread_rwlock.c

LewisR

2014-12-20 09:58

developer   ~0002948

Steve:

This is true, but I can't see how this occurs for the code with my
patches. Unless I am missing something, the calls appear to occur while
apr_file_read is looping.

Paul:

This is the case for buffered reads in apr_file_read()

There are multiple calls to apr_thread_mutex_lock/unlock, which in turn call DosRequestMutexSem/DosReleaseMutexSem

This code path is used (for example) when reading a text file from a shared drive.

Steve:

I understand, but I don't understand how this code could be running at the
time it shows up in the logs. At the time the DosRequestMutexSem is
reported, I believe that the unbuffered read code is running and that the
code is between the Dos32Read and the Dos32WaitEventSem. As th
apr_file_read patch is currently written the only thing that can happen
after Dos32Read returns ERROR_NO_DATA is that DosWaitEventSem. is called.

I guess it's possible some sort of signal event caused an exception
handler to be invoked. Perhaps enabling tracing for the exception APIs
will reveal more.

Lewis:

error_log says:

[Sat Nov 15 21:25:09 2014] [error] [client 192.168.100.8] File does not
exist: J:/APPS/apache2/htdocs/2rosenthals.net/coppermine/scripts.js [Sat
Nov 15 21:25:12 2014] [error] [client 192.168.100.8] Script timed out
before returning headers: php.exe

Steve:

This is how we fail. Httpd is reporting the fact that the event semaphore
wait timed out. The Dos32Write that wrote this message to the log is
reported in the .trc file.

Recall what apr_fileread is doing:

    for (;;) {
        int post_count;
        if (thefile->pipe)
        DosResetEventSem(thefile->pipeSem, &post_count);
        rc = DosRead(thefile->filedes, buf, *len, &nbytes);
        if (!thefile->pipe)
        break; // Blocking i/o - we are done
        if (rc == ERROR_MORE_DATA) {
        // Buffer is full - caller will read more eventually
        rc = NO_ERROR;
        break;
        }
        if (thefile->timeout == 0)
        break;
        if (thefile->timeout == 0)
        break;
        if (rc != ERROR_NO_DATA)
        break;
        rc = DosWaitEventSem(thefile->pipeSem, thefile->timeout >= 0 ?
thefile->timeout / 1000 : SEM_INDEFINITE_WAIT);
        if (rc == ERROR_TIMEOUT) {
        *len = 0;
        return APR_TIMEUP;
        }
    } // for

This code loops until DosRead returns data, DosWaitEventSem times out or
DosRead reports and error.

[...]

232 is ERROR_NO_DATA and is a normal response for a Dos32Read on a
non-blocking pipe when the writer (i.e. php.exe) is running slow relative
to the reader.

640 is ERROR_TIMEOUT and in our case this is a bad thing. It means the
writer is really slow relative to the reader. It causes the reader to
stop reading which results in the write (i.e. php.exe) hanging waiting for
Dos32Write to complete.

The reason I want to see the traces with stamping is so that I can see
when the httpd timeout occurs relative to the php Dos32Write that get
stuck.

The fix to the stuck php.exes may be as simple as adjusting the timeout
settings.

LewisR

2014-12-20 10:06

developer   ~0002949

Steve:

I recommend that we adjust the timeouts, if that is really the cause of
what we are seeing. The timeouts should be see to match "should not
occur" situations.

Paul:

fwiw, best I can tell, this is done using the timeout parameter in httpd.conf - default value is 300(!) seconds.

more at: http://httpd.apache.org/docs/2.2/mod/core.html#timeout

Lewis:

Got *1* stuck PHP process after a half dozen tries. Siege was set for
max 10 seconds between reuqests, and 2 clients. I think all together we
had 4 PHP processes fire, and one was stuck (the original, I think,
though honestly, I didn't check anything other than the timestamp. I was
able to kill the process normally from TOP.

Steve:

I looked at the zip and it confirms my analysis. To avoid the failures we
need to increase the pipe (actually the filter response) timeout. This is
controlled by the TimeOut directive. The docs claim it defaults to 300
seconds, so you must have set it to 10 seconds because that's what the
traces show.

Lewis:

I then pushed the timeout to 600 seconds (which shouldn't be necessary in any case, as the siege only lasts a minute or two), and went to 20 clients. This got me to a 72.8% availability in siege and a longest transaction time of 29.77 seconds. All PHP sessions closed, but then the VM fell over and rebooted.

I will run some more scenarios with this, but I am still concerned that
if the timeout value is too low, the worst that should happen is we
return a blank page and log an error.

Steve:

I agree. There are two things going on here. First is what httpd does
when it times out. As we know, it logs the timeout. I'm pretty sure it
just aborts the connection after this. As we have seen in the traces, the
amount of data that php.exe writes before hanging can vary. What we don't
know is how much of this httpd wrote to the client before aborting.

The second thing is what's going on on the php.exe side when httpd times
out. It appears that httpd is not really shutting down the pipe and I'm
not sure why. If the pipe was closed, I would expect the DosWrite on the
php site to fail with ERROR_BROKEN_PIPE. I suspect this is something that
needs to be tweaked in the APR.

LewisR

2014-12-20 10:12

developer   ~0002950

Lewis:

Sieging with 200 clients with a max of 1 second apart (the original
settings), got me to 59 total processes, and httpd stopped serving
requests. Prior to this (default_socket_timeout = 30), I was able to
fire almost 300 total processes, and all but 50 PHP sessions shut down.

Steve:

This makes some sense. Setting the socket timeout too low is probably
just another way to kill the httpd side of conversation before the php
side can finish writing to the pipe. The result is a hung php.

Lewis:

I can say without a doubt that performance and stability are measurably
better with php running as module than as cgi.

Steve:

Assuming one does not run out of memory in the httpd process, this would
be my expectation. OS/2 processes are expensive and slow to start
compared to Linux processes.

Lewis:

Running as module, Apache starts up with 205 threads. Once traffic hits,
the thread count will increase to somewhere over 500, and then gradually
decline, however, it does not get all the way back down to 205 (it
leaves off somewhere around 315).

Subsequent sieges gradually erode stability, until either the daemon stops serving pages or the VM reboots.

Steve:

This implies that httpd is leaking something killing it does not recover.

[Steve supplied (final) cleaned-up readwrite.c.diff (readwrite4.c.diff).]

LewisR

2014-12-20 10:12

developer  

readwrite4.c.diff (11,475 bytes)
Index: readwrite.c
===================================================================
--- readwrite.c	(revision 1629784)
+++ readwrite.c	(working copy)
@@ -24,20 +24,34 @@
 
 #include <malloc.h>
 
-APR_DECLARE(apr_status_t) apr_file_read(apr_file_t *thefile, void *buf, apr_size_t *nbytes)
+APR_DECLARE(apr_status_t) apr_file_read(apr_file_t *thefile, void *buf, apr_size_t *len)
 {
     ULONG rc = 0;
-    ULONG bytesread;
+    ULONG bytes_read = 0;
 
-    if (!thefile->isopen) {
-        *nbytes = 0;
-        return APR_EBADF;
+    if (*len <= 0) {
+        *len = 0;
+        return APR_SUCCESS;
     }
-
+#if 1
+    /* Handle the ungetchar if there is one */
+    /* copied from win32/readwrite.c */
+    if (thefile->ungetchar != -1) {
+        bytes_read = 1;
+        *(char *)buf = (char)thefile->ungetchar;
+        buf = (char *)buf + 1;
+        (*len)--;
+        thefile->ungetchar = -1;
+        if (*len == 0) {
+            *len = bytes_read;
+            return APR_SUCCESS;
+        }
+    }
+#endif
     if (thefile->buffered) {
         char *pos = (char *)buf;
         ULONG blocksize;
-        ULONG size = *nbytes;
+        ULONG size = *len;
 
         apr_thread_mutex_lock(thefile->mutex);
 
@@ -56,20 +70,20 @@
 
         while (rc == 0 && size > 0) {
             if (thefile->bufpos >= thefile->dataRead) {
-                ULONG bytesread;
+                ULONG bytes_read;
                 rc = DosRead(thefile->filedes, thefile->buffer,
-                             thefile->bufsize, &bytesread);
-
-                if (bytesread == 0) {
+                             thefile->bufsize, &bytes_read);
+                if (bytes_read == 0) {
                     if (rc == 0)
                         thefile->eof_hit = TRUE;
                     break;
                 }
-
-                thefile->dataRead = bytesread;
+		else {
+	                thefile->dataRead = bytes_read;
                 thefile->filePtr += thefile->dataRead;
                 thefile->bufpos = 0;
             }
+            }
 
             blocksize = size > thefile->dataRead - thefile->bufpos ? thefile->dataRead - thefile->bufpos : size;
             memcpy(pos, thefile->buffer + thefile->bufpos, blocksize);
@@ -78,40 +92,50 @@
             size -= blocksize;
         }
 
-        *nbytes = rc == 0 ? pos - (char *)buf : 0;
+        *len = rc == 0 ? pos - (char *)buf : 0;
         apr_thread_mutex_unlock(thefile->mutex);
 
-        if (*nbytes == 0 && rc == 0 && thefile->eof_hit) {
+        if (*len == 0 && rc == 0 && thefile->eof_hit) {
             return APR_EOF;
         }
 
         return APR_FROM_OS_ERROR(rc);
     } else {
+        /* Unbuffered i/o */
+        apr_size_t nbytes;
+
+        // 2014-11-11 SHL reworked to support ERROR_MORE_DATA etc.
+	for (;;) {
+	    int post_count;
         if (thefile->pipe)
-            DosResetEventSem(thefile->pipeSem, &rc);
-
-        rc = DosRead(thefile->filedes, buf, *nbytes, &bytesread);
-
-        if (rc == ERROR_NO_DATA && thefile->timeout != 0) {
-            int rcwait = DosWaitEventSem(thefile->pipeSem, thefile->timeout >= 0 ? thefile->timeout / 1000 : SEM_INDEFINITE_WAIT);
-
-            if (rcwait == 0) {
-                rc = DosRead(thefile->filedes, buf, *nbytes, &bytesread);
+		DosResetEventSem(thefile->pipeSem, &post_count);
+	    rc = DosRead(thefile->filedes, buf, *len, &nbytes);
+	    if (!thefile->pipe)
+		break;			// Blocking i/o - we are done
+	    if (rc == ERROR_MORE_DATA) {
+		// Buffer is full - caller will read more eventually
+		rc = NO_ERROR;
+		break;
             }
-            else if (rcwait == ERROR_TIMEOUT) {
-                *nbytes = 0;
+	    if (thefile->timeout == 0)
+		break;
+	    if (rc != ERROR_NO_DATA)
+		break;
+	    rc = DosWaitEventSem(thefile->pipeSem, thefile->timeout >= 0 ? thefile->timeout / 1000 : SEM_INDEFINITE_WAIT);
+	    if (rc == ERROR_TIMEOUT) {
+		*len = 0;
                 return APR_TIMEUP;
             }
-        }
+	} // for
 
         if (rc) {
-            *nbytes = 0;
+	    *len = 0;
             return APR_FROM_OS_ERROR(rc);
         }
 
-        *nbytes = bytesread;
+	*len = nbytes;
         
-        if (bytesread == 0) {
+	if (nbytes == 0) {
             thefile->eof_hit = TRUE;
             return APR_EOF;
         }
@@ -126,17 +150,14 @@
 {
     ULONG rc = 0;
     ULONG byteswritten;
-
     if (!thefile->isopen) {
         *nbytes = 0;
         return APR_EBADF;
     }
-
     if (thefile->buffered) {
         char *pos = (char *)buf;
         int blocksize;
         int size = *nbytes;
-
         apr_thread_mutex_lock(thefile->mutex);
 
         if ( thefile->direction == 0 ) {
@@ -147,19 +168,18 @@
             thefile->bufpos = thefile->dataRead = 0;
             thefile->direction = 1;
         }
-
         while (rc == 0 && size > 0) {
-            if (thefile->bufpos == thefile->bufsize) /* write buffer is full */
+	    if (thefile->bufpos == thefile->bufsize){ /* write buffer is full */
                 /* XXX bug; - rc is double-transformed os->apr below */
                 rc = apr_file_flush(thefile);
-
-            blocksize = size > thefile->bufsize - thefile->bufpos ? thefile->bufsize - thefile->bufpos : size;
+		}
+	    blocksize = size > thefile->bufsize - thefile->bufpos ? 
+	                       thefile->bufsize - thefile->bufpos : size;
             memcpy(thefile->buffer + thefile->bufpos, pos, blocksize);
             thefile->bufpos += blocksize;
             pos += blocksize;
             size -= blocksize;
         }
-
         apr_thread_mutex_unlock(thefile->mutex);
         return APR_FROM_OS_ERROR(rc);
     } else {
@@ -167,20 +187,16 @@
             FILELOCK all = { 0, 0x7fffffff };
             ULONG newpos;
             rc = DosSetFileLocks(thefile->filedes, NULL, &all, -1, 0);
-
             if (rc == 0) {
                 rc = DosSetFilePtr(thefile->filedes, 0, FILE_END, &newpos);
-
                 if (rc == 0) {
-                    rc = DosWrite(thefile->filedes, buf, *nbytes, &byteswritten);
+	            rc = DosWrite(thefile->filedes, (PVOID)buf, *nbytes, &byteswritten);
                 }
-
                 DosSetFileLocks(thefile->filedes, &all, NULL, -1, 0);
             }
         } else {
             rc = DosWrite(thefile->filedes, buf, *nbytes, &byteswritten);
         }
-
         if (rc) {
             *nbytes = 0;
             return APR_FROM_OS_ERROR(rc);
@@ -193,94 +209,84 @@
 
 
 
-#ifdef HAVE_WRITEV
-
 APR_DECLARE(apr_status_t) apr_file_writev(apr_file_t *thefile, const struct iovec *vec, apr_size_t nvec, apr_size_t *nbytes)
 {
-    int bytes;
+    apr_status_t rv = APR_SUCCESS;
+    apr_size_t i;
+    apr_size_t bwrote = 0;
+    char *buf;
 
-    if (thefile->buffered) {
-        apr_status_t rv = apr_file_flush(thefile);
+    *nbytes = 0;
+    for (i = 0; i < nvec; i++) {
+	buf = vec[i].iov_base;
+	bwrote = vec[i].iov_len;
+	rv = apr_file_write(thefile, buf, &bwrote);
+	*nbytes += bwrote;
         if (rv != APR_SUCCESS) {
-            return rv;
+	    break;
         }
     }
-
-    if ((bytes = writev(thefile->filedes, vec, nvec)) < 0) {
-        *nbytes = 0;
-        return errno;
+    return rv;
     }
-    else {
-        *nbytes = bytes;
-        return APR_SUCCESS;
-    }
-}
-#endif
 
 
 
 APR_DECLARE(apr_status_t) apr_file_putc(char ch, apr_file_t *thefile)
 {
-    ULONG rc;
-    ULONG byteswritten;
+    apr_size_t nbytes = 1;
 
-    if (!thefile->isopen) {
-        return APR_EBADF;
+    return apr_file_write(thefile, &ch, &nbytes);
     }
 
-    rc = DosWrite(thefile->filedes, &ch, 1, &byteswritten);
 
-    if (rc) {
-        return APR_FROM_OS_ERROR(rc);
-    }
     
-    return APR_SUCCESS;
-}
-
-
-
 APR_DECLARE(apr_status_t) apr_file_ungetc(char ch, apr_file_t *thefile)
 {
-    apr_off_t offset = -1;
-    return apr_file_seek(thefile, APR_CUR, &offset);
+    thefile->ungetchar = (unsigned char)ch;
+    return APR_SUCCESS; 
 }
 
 
 APR_DECLARE(apr_status_t) apr_file_getc(char *ch, apr_file_t *thefile)
 {
-    ULONG rc;
-    apr_size_t bytesread;
+    apr_size_t nbytes = 1;
 
-    if (!thefile->isopen) {
-        return APR_EBADF;
+    return apr_file_read(thefile, ch, &nbytes);
+
     }
 
-    bytesread = 1;
-    rc = apr_file_read(thefile, ch, &bytesread);
 
-    if (rc) {
-        return rc;
-    }
     
-    if (bytesread == 0) {
-        thefile->eof_hit = TRUE;
-        return APR_EOF;
+APR_DECLARE(apr_status_t) apr_file_puts(const char *str, apr_file_t *thefile)
+{
+    return apr_file_write_full(thefile, str, strlen(str), NULL);
     }
     
-    return APR_SUCCESS;
-}
 
 
-
-APR_DECLARE(apr_status_t) apr_file_puts(const char *str, apr_file_t *thefile)
+apr_status_t apr_file_flush_locked(apr_file_t *thefile)
 {
-    apr_size_t len;
+    apr_status_t rv = APR_SUCCESS;
 
-    len = strlen(str);
-    return apr_file_write(thefile, str, &len); 
+    if (thefile->direction == 1 && thefile->bufpos) {
+	apr_ssize_t written;
+
+	do {
+	    written = write(thefile->filedes, thefile->buffer, thefile->bufpos);
+	} while (written == -1 && errno == EINTR);
+	if (written == -1) {
+	    rv = errno;
+	} else {
+	    thefile->filePtr += written;
+	    thefile->bufpos = 0;
 }
+    }
 
+    return rv;
+}
 
+
+
 APR_DECLARE(apr_status_t) apr_file_flush(apr_file_t *thefile)
 {
     if (thefile->buffered) {
@@ -349,10 +355,61 @@
 }
 
 
+/* Pull from unix code to start a sync up */
+#if (defined(__INNOTEK_LIBC__) || defined(__WATCOMC__) )
 
+struct apr_file_printf_data {
+    apr_vformatter_buff_t vbuff;
+    apr_file_t *fptr;
+    char *buf;
+};
+
+static int file_printf_flush(apr_vformatter_buff_t *buff)
+{
+    struct apr_file_printf_data *data = (struct apr_file_printf_data *)buff;
+
+    if (apr_file_write_full(data->fptr, data->buf,
+	                    data->vbuff.curpos - data->buf, NULL)) {
+	return -1;
+    }
+
+    data->vbuff.curpos = data->buf;
+    return 0;
+}
+
 APR_DECLARE_NONSTD(int) apr_file_printf(apr_file_t *fptr, 
                                         const char *format, ...)
 {
+    struct apr_file_printf_data data;
+    va_list ap;
+    int count;
+
+    /* don't really need a HUGE_STRING_LEN anymore */
+    data.buf = _lmalloc(HUGE_STRING_LEN);
+    if (data.buf == NULL) {
+	return -1;
+    }
+    data.vbuff.curpos = data.buf;
+    data.vbuff.endpos = data.buf + HUGE_STRING_LEN;
+    data.fptr = fptr;
+    va_start(ap, format);
+    count = apr_vformatter(file_printf_flush,
+	                   (apr_vformatter_buff_t *)&data, format, ap);
+    /* apr_vformatter does not call flush for the last bits */
+    if (count >= 0) file_printf_flush((apr_vformatter_buff_t *)&data);
+
+    va_end(ap);
+
+    free(data.buf);
+
+    return count;
+}
+
+#else
+
+APR_DECLARE_NONSTD(int) apr_file_printf(apr_file_t *fptr, 
+	                                const char *format, ...)
+{
     int cc;
     va_list ap;
     char *buf;
@@ -369,9 +426,9 @@
     free(buf);
     return (cc == APR_SUCCESS) ? len : -1;
 }
+#endif
 
 
-
 apr_status_t apr_file_check_read(apr_file_t *fd)
 {
     int rc;
readwrite4.c.diff (11,475 bytes)

LewisR

2014-12-20 10:19

developer   ~0002951

Next phase...

Steve:

I think that the results of the siege testing failure have exposed a subte
error in how the apr closes pipe handles.

The docs have this to say about what happens when the server closes the
pipe handle:

"If the server end closes when the client end is already closed, the pipe
is deallocated immediately; otherwise, the pipe is not deallocated until
the last client handle is closed."

In the context of our CGI testing, when there are no timeouts, the 1st
case applies. The php side closes its side of the pipe and when the httpd
closes its side of the pipe, the pipe resources are deallocated. I assume
this includes the pipe semaphore.

When we get a timeout, the 2nd case applies. From the php client's POV
the pipe is still open, so the write never completes and the php client
hangs around forever.

To avoid this, the server needs to disconnect from the named pipe before
closing the handle. The will cause the write to fail with
ERROR_BROKEN_PIPE. It the client is gone, as is the normal case, the
disconnect should no effect.

What should work is inserting the following in apr_file_close

  if (file->pipe == 1) {
    DosCloseEventSem(file->pipeSem);
    DosDisconnectNPipe(file->filedes);
  }

before

  rc = DosClose(file->filedes);

There's no need to check for errors.

We only want to disconnect the pipe handle, not the client file handle, so
we need to modify

pipe.c:112
    (*out)->pipe = 1;

to read
    (*out)->pipe = 2; // 2014-11-17 SHL mark as client side of pipe

What I don't yet know yet is if we really need to close the pipe
semaphore. This may be done automatically when we close the pipe handle.
It depends on what the docs mean by pipe resources. IAC, we can verify
this later with a bit of testing.

[...]

FWIW, I went poking about the internet for more code that uses
the APIs in question. There's not much, but I did find

  http://www.virtualbox.org/svn/vbox/trunk/src/VBox/Runtime/r3/os2/pipe-os2.cpp

which is probably by Knut. This code explicitly does not disconnect, but
that's probably because it closes both sides of the pipe itself.

I also found:

http://fpc.sourcearchive.com/documentation/2.2.4/os2_2simpleipc_8inc-source.html

which implements the close as I am proposing.

[...]

Another though just occurred to me. Thread ordinals get reused, so this
means that pipe names get reused to. Apache assumes that when a thread
terminates, the associated named pipe is gone too. However, when we have
a hanging php instance, this might not be true.

Paul [providing a new build with the proposed mod]:

http://smedley.id.au/tmp/httpd-dll-20141118.zip

[diffs of Steve's proposed mods attached as open.c.diff and pipe.c.diff]

LewisR

2014-12-20 10:20

developer  

open.c.diff (4,577 bytes)
Index: open.c
===================================================================
--- open.c	(revision 1629784)
+++ open.c	(working copy)
@@ -28,14 +28,70 @@
     return apr_file_close(file);
 }
 
+#ifdef __INNOTEK_LIBC__
+static apr_status_t file_cleanup(apr_file_t *file)
+{
+    apr_status_t rv = APR_SUCCESS;
 
+    if (close(file->filedes) == 0) {
+        file->filedes = -1;
+        if (file->flags & APR_DELONCLOSE) {
+            unlink(file->fname);
+        }
+#if APR_HAS_THREADS
+        if (file->mutex) {
+            rv = apr_thread_mutex_destroy(file->mutex);
+        }
+#endif
+    }
+    else {
+        /* Are there any error conditions other than EINTR or EBADF? */
+        rv = errno;
+    }
+#ifndef WAITIO_USES_POLL
+    if (file->pollset != NULL) {
+        int pollset_rv = apr_pollset_destroy(file->pollset);
+        /* If the file close failed, return its error value,
+         * not apr_pollset_destroy()'s.
+         */
+        if (rv == APR_SUCCESS) {
+            rv = pollset_rv;
+        }
+    }
+#endif /* !WAITIO_USES_POLL */
+    return rv;
+}
 
+apr_status_t apr_unix_file_cleanup(void *thefile)
+{
+    apr_file_t *file = thefile;
+    apr_status_t flush_rv = APR_SUCCESS, rv = APR_SUCCESS;
+
+    if (file->buffered) {
+        flush_rv = apr_file_flush(file);
+    }
+
+    rv = file_cleanup(file);
+
+    return rv != APR_SUCCESS ? rv : flush_rv;
+}
+
+apr_status_t apr_unix_child_file_cleanup(void *thefile)
+{
+    return file_cleanup(thefile);
+}
+#endif
+
+
 APR_DECLARE(apr_status_t) apr_file_open(apr_file_t **new, const char *fname, apr_int32_t flag,  apr_fileperms_t perm, apr_pool_t *pool)
 {
     int oflags = 0;
     int mflags = OPEN_FLAGS_FAIL_ON_ERROR|OPEN_SHARE_DENYNONE|OPEN_FLAGS_NOINHERIT;
     int rv;
     ULONG action;
+    ULONG    CurMaxFH      = 0;          /* Current count of handles         */
+    LONG     ReqCount      = 0;          /* Number to adjust file handles    */
+
     apr_file_t *dafile = (apr_file_t *)apr_palloc(pool, sizeof(apr_file_t));
 
     if (flag & APR_FOPEN_NONBLOCK) {
@@ -90,6 +146,15 @@
     
     rv = DosOpen(fname, &(dafile->filedes), &action, 0, 0, oflags, mflags, NULL);
     
+    if (rv == ERROR_TOO_MANY_OPEN_FILES) {
+	rv = DosSetRelMaxFH(&ReqCount,     /* Using 0 here will return the       */
+                      &CurMaxFH);    /* current number of file handles     */
+	ReqCount      = 10L;         /* Want 10 more file handles         */
+	rv = DosSetRelMaxFH(&ReqCount,&CurMaxFH);     /* Change handle maximum */
+
+	rv = DosOpen(fname, &(dafile->filedes), &action, 0, 0, oflags, mflags, NULL);
+    }    
+
     if (rv == 0 && (flag & APR_FOPEN_APPEND)) {
         ULONG newptr;
         rv = DosSetFilePtr(dafile->filedes, 0, FILE_END, &newptr );
@@ -108,7 +173,9 @@
     dafile->dataRead = 0;
     dafile->direction = 0;
     dafile->pipe = FALSE;
+    dafile->ungetchar = -1;
 
+
     if (!(flag & APR_FOPEN_NOCLEANUP)) {
         apr_pool_cleanup_register(dafile->pool, dafile, apr_file_cleanup, apr_file_cleanup);
     }
@@ -127,6 +194,11 @@
     if (file && file->isopen) {
         /* XXX: flush here is not mutex protected */
         status = apr_file_flush(file);
+	/* 2014-11-18 SHL ensure client sees ERROR_BROKEN_PIPE */
+	if (file->pipe == 1) {
+	  DosCloseEventSem(file->pipeSem);
+	  DosDisconnectNPipe(file->filedes);
+	}
         rc = DosClose(file->filedes);
     
         if (rc == 0) {
@@ -197,6 +269,7 @@
     (*file)->flags = flags;
     (*file)->pipe = FALSE;
     (*file)->buffered = (flags & APR_FOPEN_BUFFERED) > 0;
+    (*file)->ungetchar = -1;
 
     if ((*file)->buffered) {
         apr_status_t rv;
@@ -279,11 +352,9 @@
     ULONG state;
 
     rv = DosQueryFHState(thefile->filedes, &state);
-
     if (rv == 0 && (state & OPEN_FLAGS_NOINHERIT) != 0) {
         rv = DosSetFHState(thefile->filedes, state & ~OPEN_FLAGS_NOINHERIT);
     }
-
     return APR_FROM_OS_ERROR(rv);
 }
 
@@ -295,10 +366,14 @@
     ULONG state;
 
     rv = DosQueryFHState(thefile->filedes, &state);
-
     if (rv == 0 && (state & OPEN_FLAGS_NOINHERIT) == 0) {
-        rv = DosSetFHState(thefile->filedes, state | OPEN_FLAGS_NOINHERIT);
+/*        rv = DosSetFHState(thefile->filedes, state | OPEN_FLAGS_NOINHERIT);*/
     }
-
     return APR_FROM_OS_ERROR(rv);
 }
+
+APR_DECLARE(apr_status_t) apr_file_link(const char *from_path, 
+					  const char *to_path)
+{
+    return APR_ENOTIMPL;;
+}
open.c.diff (4,577 bytes)

LewisR

2014-12-20 10:20

developer  

pipe.c.diff (3,450 bytes)
Index: pipe.c
===================================================================
--- pipe.c	(revision 1629784)
+++ pipe.c	(working copy)
@@ -30,6 +30,8 @@
     ULONG rc, action;
     static int id = 0;
     char pipename[50];
+    ULONG    CurMaxFH      = 0;          /* Current count of handles         */
+    LONG     ReqCount      = 0;          /* Number to adjust file handles    */
 
     sprintf(pipename, "/pipe/%d.%d", getpid(), id++);
     rc = DosCreateNPipe(pipename, filedes, NP_ACCESS_INBOUND, NP_NOWAIT|1, 4096, 4096, 0);
@@ -48,7 +50,19 @@
                   OPEN_ACTION_OPEN_IF_EXISTS | OPEN_ACTION_FAIL_IF_NEW,
                   OPEN_ACCESS_WRITEONLY | OPEN_SHARE_DENYREADWRITE,
                   NULL);
+    if (rc == ERROR_TOO_MANY_OPEN_FILES) {
+	rc = DosSetRelMaxFH(&ReqCount,     /* Using 0 here will return the       */
+                      &CurMaxFH);    /* current number of file handles     */
+	ReqCount      = 10L;         /* Want 10 more file handles         */
+	rc = DosSetRelMaxFH(&ReqCount,&CurMaxFH);     /* Change handle maximum */
 
+	rc = DosOpen (pipename, filedes+1, &action, 0, FILE_NORMAL,
+                  OPEN_ACTION_OPEN_IF_EXISTS | OPEN_ACTION_FAIL_IF_NEW,
+                  OPEN_ACCESS_WRITEONLY | OPEN_SHARE_DENYREADWRITE,
+                  NULL);
+
+    }    
+
     if (rc) {
         DosClose(filedes[0]);
         return APR_FROM_OS_ERROR(rc);
@@ -84,6 +98,7 @@
     (*in)->flags = 0;
     (*in)->pipe = 1;
     (*in)->timeout = -1;
+    (*in)->ungetchar = -1;
     (*in)->blocking = BLK_ON;
     apr_pool_cleanup_register(pool, *in, apr_file_cleanup, apr_pool_cleanup_null);
 
@@ -94,7 +109,8 @@
     (*out)->isopen = TRUE;
     (*out)->buffered = FALSE;
     (*out)->flags = 0;
-    (*out)->pipe = 1;
+    (*out)->pipe = 2;			// 2014-11-17 SHL mark as client pipe
+    (*out)->ungetchar = -1;
     (*out)->timeout = -1;
     (*out)->blocking = BLK_ON;
     apr_pool_cleanup_register(pool, *out, apr_file_cleanup, apr_pool_cleanup_null);
@@ -143,24 +159,25 @@
 
 APR_DECLARE(apr_status_t) apr_file_pipe_timeout_set(apr_file_t *thepipe, apr_interval_time_t timeout)
 {
+    int rc = 0;
+
     if (thepipe->pipe == 1) {
         thepipe->timeout = timeout;
 
-        if (thepipe->timeout >= 0) {
-            if (thepipe->blocking != BLK_OFF) {
+        if ((thepipe->timeout >= 0) && (thepipe->blocking != BLK_OFF) ) {
                 thepipe->blocking = BLK_OFF;
-                return APR_FROM_OS_ERROR(DosSetNPHState(thepipe->filedes, NP_NOWAIT));
+            rc = DosSetNPHState(thepipe->filedes, NP_NOWAIT);
             }
-        }
-        else if (thepipe->timeout == -1) {
-            if (thepipe->blocking != BLK_ON) {
+        else if ( (thepipe->timeout == -1) && (thepipe->blocking != BLK_ON) ) {
                 thepipe->blocking = BLK_ON;
-                return APR_FROM_OS_ERROR(DosSetNPHState(thepipe->filedes, NP_WAIT));
+            rc = DosSetNPHState(thepipe->filedes, NP_WAIT);
             }
+
+        return APR_FROM_OS_ERROR(rc);
+
+    } else return APR_EINVAL;
+
         }
-    }
-    return APR_EINVAL;
-}
 
 
 
@@ -186,6 +203,7 @@
     (*file)->pipe = 1;
     (*file)->blocking = BLK_UNKNOWN; /* app needs to make a timeout call */
     (*file)->timeout = -1;
+    (*file)->ungetchar = -1;
     (*file)->filedes = *thefile;
 
     if (register_cleanup) {
pipe.c.diff (3,450 bytes)

LewisR

2014-12-20 10:26

developer   ~0002952

Lewis:

I've trimmed the httpd timeout from 600 seconds to 90 seconds (all php
timeouts - socket, mysql, and script execution are set to 60 seconds).

Steve:

I can't make the timeouts trigger here even with Timeout set to 1 second.
What I've done is add a 30 second sleep to the php code to ensure it does
not complete in anything like a timely manner. Very odd.

[...]

I starting looking at pdump.000 from

  arcanoae-server-dump-after-sixth-siege-2014-11-18-all.zip

As expected all the httpd threads are blocked. I need to do more work to
figure out if all the threads are waiting on the same semaphore or
whatever.

[testing left off 2014-11-18 on this system]

psmedley

2015-01-02 07:32

administrator   ~0002985

Hey Lewis, can you please post here the siege parameters you're using for testing?

LewisR

2015-01-02 07:48

developer   ~0002986

Surely, Paul.

Benchmarking is done with:

siege -b -t60S http://www.2rosenthals.net/

The above url hits index.html, which contains pure HTML.

Stress testing is done with:

siege -d 5 -c 200 -f siegeurls1

siegeurls1 is a listing of collected urls from http://www.2rosenthals.net (attached)

LewisR

2015-01-02 07:48

developer  

siegeurls1 (78,190 bytes)
http://www.2rosenthals.net/
http://www.2rosenthals.net/coppermine
http://www.2rosenthals.net/coppermine/
http://www.2rosenthals.net/coppermine/albums/upload/BVC/thumb_img011.jpg
http://www.2rosenthals.net/coppermine/albums/upload/BVC/thumb_img012.jpg
http://www.2rosenthals.net/coppermine/albums/upload/BVC/thumb_img01.jpg
http://www.2rosenthals.net/coppermine/albums/upload/BVC/thumb_img04.jpg
http://www.2rosenthals.net/coppermine/albums/upload/mjones/thumb_dsc00197.jpg
http://www.2rosenthals.net/coppermine/albums/upload/wallpapers/SuSE/9_3/thumb_suse-default.png
http://www.2rosenthals.net/coppermine/albums/userpics/10001/thumb_5192245.jpg
http://www.2rosenthals.net/coppermine/albums/userpics/10001/thumb_Dscn4596.jpg
http://www.2rosenthals.net/coppermine/albums/userpics/10001/thumb_Dscn4597.jpg
http://www.2rosenthals.net/coppermine/albums/userpics/10001/thumb_IMG00285.jpg
http://www.2rosenthals.net/coppermine/albums/userpics/10002/thumb_71407_06.jpg
http://www.2rosenthals.net/coppermine/albums/userpics/10002/thumb_71407_07.jpg
http://www.2rosenthals.net/coppermine/albums/userpics/10002/thumb_71407_08.jpg
http://www.2rosenthals.net/coppermine/albums/userpics/10002/thumb_71407_09.jpg
http://www.2rosenthals.net/coppermine/albums/userpics/10002/thumb_71407_10.jpg
http://www.2rosenthals.net/coppermine/albums/userpics/10002/thumb_71407_11.jpg
http://www.2rosenthals.net/coppermine/albums/userpics/10003/thumb_1963_Bonneville.jpg
http://www.2rosenthals.net/coppermine/albums/userpics/10003/thumb_64477956741_3300.jpg
http://www.2rosenthals.net/coppermine/albums/userpics/10003/thumb_64479166213_3300.jpg
http://www.2rosenthals.net/coppermine/albums/userpics/10003/thumb_bcar001.jpg
http://www.2rosenthals.net/coppermine/albums/userpics/10003/thumb_bcar006.jpg
http://www.2rosenthals.net/coppermine/albums/userpics/10003/thumb_bealtonmap.jpg
http://www.2rosenthals.net/coppermine/albums/userpics/10003/thumb_DSCN2749.JPG
http://www.2rosenthals.net/coppermine/albums/userpics/10003/thumb_DSCN2751.JPG
http://www.2rosenthals.net/coppermine/albums/userpics/10003/thumb_DSCN2755.JPG
http://www.2rosenthals.net/coppermine/albums/userpics/10003/thumb_dscn2824.jpg
http://www.2rosenthals.net/coppermine/albums/userpics/10003/thumb_DSCN2846.JPG
http://www.2rosenthals.net/coppermine/albums/userpics/10003/thumb_DSCN2847.JPG
http://www.2rosenthals.net/coppermine/contact.php?referer=index.php
http://www.2rosenthals.net/coppermine/contact.php?referer=index.php%3Fcat%3D1
http://www.2rosenthals.net/coppermine/contact.php?referer=index.php%3Fcat%3D13
http://www.2rosenthals.net/coppermine/contact.php?referer=index.php%3Fcat%3D14
http://www.2rosenthals.net/coppermine/contact.php?referer=index.php%3Fcat%3D15
http://www.2rosenthals.net/coppermine/contact.php?referer=index.php%3Fcat%3D16
http://www.2rosenthals.net/coppermine/contact.php?referer=index.php%3Fcat%3D17
http://www.2rosenthals.net/coppermine/contact.php?referer=index.php%3Fcat%3D4
http://www.2rosenthals.net/coppermine/contact.php?referer=index.php%3Fcat%3D6
http://www.2rosenthals.net/coppermine/contact.php?referer=index.php%3Fcat%3D9
http://www.2rosenthals.net/coppermine/contact.php?referer=index.php%3Flang%3Dxxx
http://www.2rosenthals.net/coppermine/contact.php?referer=index.php&lang=xxx
http://www.2rosenthals.net/coppermine/contact.php?referer=login.php%3Freferer%3Dindex.php
http://www.2rosenthals.net/coppermine/contact.php?referer=search.php
http://www.2rosenthals.net/coppermine/contact.php?referer=thumbnails.php%3Falbum%3D15
http://www.2rosenthals.net/coppermine/contact.php?referer=thumbnails.php%3Falbum%3D18
http://www.2rosenthals.net/coppermine/contact.php?referer=thumbnails.php%3Falbum%3D2
http://www.2rosenthals.net/coppermine/contact.php?referer=thumbnails.php%3Falbum%3D22
http://www.2rosenthals.net/coppermine/contact.php?referer=thumbnails.php%3Falbum%3D24
http://www.2rosenthals.net/coppermine/contact.php?referer=thumbnails.php%3Falbum%3D5
http://www.2rosenthals.net/coppermine/contact.php?referer=thumbnails.php%3Falbum%3D6
http://www.2rosenthals.net/coppermine/contact.php?referer=thumbnails.php%3Falbum%3D7
http://www.2rosenthals.net/coppermine/displayimage.php?album=15&pid=217
http://www.2rosenthals.net/coppermine/displayimage.php?album=15&pid=236
http://www.2rosenthals.net/coppermine/displayimage.php?album=15&pid=237
http://www.2rosenthals.net/coppermine/displayimage.php?album=15&pid=238
http://www.2rosenthals.net/coppermine/displayimage.php?album=15&pid=306
http://www.2rosenthals.net/coppermine/displayimage.php?album=15&pid=308
http://www.2rosenthals.net/coppermine/displayimage.php?album=15&pid=315
http://www.2rosenthals.net/coppermine/displayimage.php?album=16&pid=186
http://www.2rosenthals.net/coppermine/displayimage.php?album=18&pid=189
http://www.2rosenthals.net/coppermine/displayimage.php?album=18&pid=344
http://www.2rosenthals.net/coppermine/displayimage.php?album=18&pid=345
http://www.2rosenthals.net/coppermine/displayimage.php?album=18&pid=348
http://www.2rosenthals.net/coppermine/displayimage.php?album=18&pid=349
http://www.2rosenthals.net/coppermine/displayimage.php?album=18&pid=353
http://www.2rosenthals.net/coppermine/displayimage.php?album=22&pid=562
http://www.2rosenthals.net/coppermine/displayimage.php?album=7&pid=17
http://www.2rosenthals.net/coppermine/displayimage.php?album=7&pid=18
http://www.2rosenthals.net/coppermine/displayimage.php?album=7&pid=564
http://www.2rosenthals.net/coppermine/displayimage.php?album=7&pid=565
http://www.2rosenthals.net/coppermine/help.php?f=empty.htm&base=64&h=czoyOToiVmlzdWFsIGNvbmZpcm1hdGlvbiAoY2FwdGNoYSkiOw%3D%3D&t=czoxOTE6IlRvIGF2b2lkIHNwYW0sIHlvdSBoYXZlIHRvIGNvbmZpcm0gdGhhdCB5b3UgYXJlIGFuIGFjdHVhbCBodW1hbiBiZWluZyBhbmQgbm90IGp1c3QgYSBib3Qgc2NyaXB0IGJ5IGVudGVyaW5nIHRoZSBkaXNwbGF5ZWQgdGV4dC48YnIgLz5DYXBpdGFsaXphdGlvbiBkb2VzIG5vdCBtYXR0ZXIsIHlvdSBjYW4gdHlwZSBpbiBsb3dlcmNhc2UuIjs%3D
http://www.2rosenthals.net/coppermine/images/flags/reset.png
http://www.2rosenthals.net/coppermine/images/help.gif
http://www.2rosenthals.net/coppermine/images/h_powered-mysql.gif
http://www.2rosenthals.net/coppermine/images/h_powered-php.gif
http://www.2rosenthals.net/coppermine/images/h_valid-css.gif
http://www.2rosenthals.net/coppermine/images/h_valid-xhtml10.gif
http://www.2rosenthals.net/coppermine/images/powered-mysql.gif
http://www.2rosenthals.net/coppermine/images/powered-php.gif
http://www.2rosenthals.net/coppermine/images/spacer.gif
http://www.2rosenthals.net/coppermine/images/thumbs/thumb_avi.png
http://www.2rosenthals.net/coppermine/images/thumbs/thumb_private.png
http://www.2rosenthals.net/coppermine/images/valid-css.gif
http://www.2rosenthals.net/coppermine/images/valid-xhtml10.gif
http://www.2rosenthals.net/coppermine/index.php
http://www.2rosenthals.net/coppermine/index.php?cat=0
http://www.2rosenthals.net/coppermine/index.php?cat=1
http://www.2rosenthals.net/coppermine/index.php?cat=10
http://www.2rosenthals.net/coppermine/index.php?cat=10007
http://www.2rosenthals.net/coppermine/index.php?cat=10&lang=xxx
http://www.2rosenthals.net/coppermine/index.php?cat=11
http://www.2rosenthals.net/coppermine/index.php?cat=11&lang=xxx
http://www.2rosenthals.net/coppermine/index.php?cat=13
http://www.2rosenthals.net/coppermine/index.php?cat=13&lang=xxx
http://www.2rosenthals.net/coppermine/index.php?cat=14
http://www.2rosenthals.net/coppermine/index.php?cat=14&lang=xxx
http://www.2rosenthals.net/coppermine/index.php?cat=15
http://www.2rosenthals.net/coppermine/index.php?cat=16
http://www.2rosenthals.net/coppermine/index.php?cat=17
http://www.2rosenthals.net/coppermine/index.php?cat=17&lang=xxx
http://www.2rosenthals.net/coppermine/index.php?cat=2
http://www.2rosenthals.net/coppermine/index.php?cat=2&lang=xxx
http://www.2rosenthals.net/coppermine/index.php?cat=3
http://www.2rosenthals.net/coppermine/index.php?cat=3&lang=xxx
http://www.2rosenthals.net/coppermine/index.php?cat=4
http://www.2rosenthals.net/coppermine/index.php?cat=4&lang=xxx
http://www.2rosenthals.net/coppermine/index.php?cat=5
http://www.2rosenthals.net/coppermine/index.php?cat=6
http://www.2rosenthals.net/coppermine/index.php?cat=6&lang=xxx
http://www.2rosenthals.net/coppermine/index.php?cat=7
http://www.2rosenthals.net/coppermine/index.php?cat=7&lang=xxx
http://www.2rosenthals.net/coppermine/index.php?cat=8
http://www.2rosenthals.net/coppermine/index.php?cat=8&lang=xxx
http://www.2rosenthals.net/coppermine/index.php?cat=9
http://www.2rosenthals.net/coppermine/index.php?lang=xxx
http://www.2rosenthals.net/coppermine/js/contact.js
http://www.2rosenthals.net/coppermine/js/jquery-1.3.2.js
http://www.2rosenthals.net/coppermine/js/jquery.elastic.js
http://www.2rosenthals.net/coppermine/js/jquery.greybox.js
http://www.2rosenthals.net/coppermine/js/scripts.js
http://www.2rosenthals.net/coppermine/js/thumbnails.js
http://www.2rosenthals.net/coppermine/login.php
http://www.2rosenthals.net/coppermine/login.php?force_login=1&referer=displayimage.php%3Falbum%3D15%26pid%3D179
http://www.2rosenthals.net/coppermine/login.php?force_login=1&referer=displayimage.php%3Falbum%3D15%26pid%3D180
http://www.2rosenthals.net/coppermine/login.php?force_login=1&referer=displayimage.php%3Falbum%3D15%26pid%3D217
http://www.2rosenthals.net/coppermine/login.php?force_login=1&referer=displayimage.php%3Falbum%3D15%26pid%3D236
http://www.2rosenthals.net/coppermine/login.php?force_login=1&referer=displayimage.php%3Falbum%3D15%26pid%3D237
http://www.2rosenthals.net/coppermine/login.php?force_login=1&referer=displayimage.php%3Falbum%3D15%26pid%3D238
http://www.2rosenthals.net/coppermine/login.php?force_login=1&referer=displayimage.php%3Falbum%3D15%26pid%3D306
http://www.2rosenthals.net/coppermine/login.php?force_login=1&referer=displayimage.php%3Falbum%3D15%26pid%3D307
http://www.2rosenthals.net/coppermine/login.php?force_login=1&referer=displayimage.php%3Falbum%3D15%26pid%3D314
http://www.2rosenthals.net/coppermine/login.php?force_login=1&referer=displayimage.php%3Falbum%3D16%26pid%3D185
http://www.2rosenthals.net/coppermine/login.php?force_login=1&referer=displayimage.php%3Falbum%3D16%26pid%3D186
http://www.2rosenthals.net/coppermine/login.php?force_login=1&referer=displayimage.php%3Falbum%3D18%26pid%3D189
http://www.2rosenthals.net/coppermine/login.php?force_login=1&referer=displayimage.php%3Falbum%3D18%26pid%3D203
http://www.2rosenthals.net/coppermine/login.php?force_login=1&referer=displayimage.php%3Falbum%3D18%26pid%3D344
http://www.2rosenthals.net/coppermine/login.php?force_login=1&referer=displayimage.php%3Falbum%3D18%26pid%3D349
http://www.2rosenthals.net/coppermine/login.php?force_login=1&referer=displayimage.php%3Falbum%3D18%26pid%3D353
http://www.2rosenthals.net/coppermine/login.php?force_login=1&referer=displayimage.php%3Falbum%3D22%26pid%3D562
http://www.2rosenthals.net/coppermine/login.php?force_login=1&referer=displayimage.php%3Falbum%3D7%26pid%3D16
http://www.2rosenthals.net/coppermine/login.php?force_login=1&referer=displayimage.php%3Falbum%3D7%26pid%3D17
http://www.2rosenthals.net/coppermine/login.php?force_login=1&referer=displayimage.php%3Falbum%3D7%26pid%3D20
http://www.2rosenthals.net/coppermine/login.php?force_login=1&referer=displayimage.php%3Falbum%3D7%26pid%3D21
http://www.2rosenthals.net/coppermine/login.php?force_login=1&referer=displayimage.php%3Falbum%3D7%26pid%3D563
http://www.2rosenthals.net/coppermine/login.php?force_login=1&referer=displayimage.php%3Falbum%3D7%26pid%3D564
http://www.2rosenthals.net/coppermine/login.php?force_login=1&referer=displayimage.php%3Falbum%3D7%26pid%3D566
http://www.2rosenthals.net/coppermine/login.php?referer=contact.php%3Freferer%3Dindex.php
http://www.2rosenthals.net/coppermine/login.php?referer=index.php
http://www.2rosenthals.net/coppermine/login.php?referer=index.php%3Fcat%3D0
http://www.2rosenthals.net/coppermine/login.php?referer=index.php%3Fcat%3D1
http://www.2rosenthals.net/coppermine/login.php?referer=index.php%3Fcat%3D10
http://www.2rosenthals.net/coppermine/login.php?referer=index.php%3Fcat%3D11
http://www.2rosenthals.net/coppermine/login.php?referer=index.php%3Fcat%3D14
http://www.2rosenthals.net/coppermine/login.php?referer=index.php%3Fcat%3D16
http://www.2rosenthals.net/coppermine/login.php?referer=index.php%3Fcat%3D2
http://www.2rosenthals.net/coppermine/login.php?referer=index.php%3Fcat%3D4
http://www.2rosenthals.net/coppermine/login.php?referer=index.php%3Fcat%3D5
http://www.2rosenthals.net/coppermine/login.php?referer=index.php%3Fcat%3D8
http://www.2rosenthals.net/coppermine/login.php?referer=index.php%3Fcat%3D9
http://www.2rosenthals.net/coppermine/login.php?referer=index.php%3Flang%3Dxxx
http://www.2rosenthals.net/coppermine/login.php?referer=index.php&lang=xxx
http://www.2rosenthals.net/coppermine/login.php?referer=thumbnails.php%3Falbum%3D16
http://www.2rosenthals.net/coppermine/login.php?referer=thumbnails.php%3Falbum%3D17
http://www.2rosenthals.net/coppermine/login.php?referer=thumbnails.php%3Falbum%3D2
http://www.2rosenthals.net/coppermine/login.php?referer=thumbnails.php%3Falbum%3D6
http://www.2rosenthals.net/coppermine/login.php?referer=thumbnails.php%3Falbum%3D7
http://www.2rosenthals.net/coppermine/plugins/light_box/slideshow/css/lightbox.css
http://www.2rosenthals.net/coppermine/plugins/light_box/slideshow/images/blank.gif
http://www.2rosenthals.net/coppermine/plugins/light_box/slideshow/images/nextlabel.gif
http://www.2rosenthals.net/coppermine/plugins/light_box/slideshow/images/prevlabel.gif
http://www.2rosenthals.net/coppermine/plugins/light_box/slideshow/js/lightbox_s.js
http://www.2rosenthals.net/coppermine/plugins/light_box/slideshow/js/prototype.js
http://www.2rosenthals.net/coppermine/plugins/light_box/slideshow/js/scriptaculous.js?load=effects
http://www.2rosenthals.net/coppermine/profile.php?uid=7
http://www.2rosenthals.net/coppermine/js/scripts.js
http://www.2rosenthals.net/coppermine/search.php
http://www.2rosenthals.net/coppermine/send_activation.php
http://www.2rosenthals.net/coppermine/themes/classic/images/button_bg_anim.gif
http://www.2rosenthals.net/coppermine/themes/classic/images/button_bg.gif
http://www.2rosenthals.net/coppermine/themes/classic/images/site_logo.png
http://www.2rosenthals.net/coppermine/themes/classic/style.css
http://www.2rosenthals.net/coppermine/thumbnails.php?album=1
http://www.2rosenthals.net/coppermine/thumbnails.php?album=10
http://www.2rosenthals.net/coppermine/thumbnails.php?album=11
http://www.2rosenthals.net/coppermine/thumbnails.php?album=12
http://www.2rosenthals.net/coppermine/thumbnails.php?album=13
http://www.2rosenthals.net/coppermine/thumbnails.php?album=13&lang=xxx
http://www.2rosenthals.net/coppermine/thumbnails.php?album=14
http://www.2rosenthals.net/coppermine/thumbnails.php?album=15
http://www.2rosenthals.net/coppermine/thumbnails.php?album=15&lang=xxx
http://www.2rosenthals.net/coppermine/thumbnails.php?album=15&page=2
http://www.2rosenthals.net/coppermine/thumbnails.php?album=15&page=6
http://www.2rosenthals.net/coppermine/thumbnails.php?album=16
http://www.2rosenthals.net/coppermine/thumbnails.php?album=17
http://www.2rosenthals.net/coppermine/thumbnails.php?album=17&lang=xxx
http://www.2rosenthals.net/coppermine/thumbnails.php?album=18
http://www.2rosenthals.net/coppermine/thumbnails.php?album=19
http://www.2rosenthals.net/coppermine/thumbnails.php?album=2
http://www.2rosenthals.net/coppermine/thumbnails.php?album=21
http://www.2rosenthals.net/coppermine/thumbnails.php?album=22
http://www.2rosenthals.net/coppermine/thumbnails.php?album=22&lang=xxx
http://www.2rosenthals.net/coppermine/thumbnails.php?album=24
http://www.2rosenthals.net/coppermine/thumbnails.php?album=24&lang=xxx
http://www.2rosenthals.net/coppermine/thumbnails.php?album=2&lang=xxx
http://www.2rosenthals.net/coppermine/thumbnails.php?album=4
http://www.2rosenthals.net/coppermine/thumbnails.php?album=5
http://www.2rosenthals.net/coppermine/thumbnails.php?album=5&lang=xxx
http://www.2rosenthals.net/coppermine/thumbnails.php?album=6
http://www.2rosenthals.net/coppermine/thumbnails.php?album=6&lang=xxx
http://www.2rosenthals.net/coppermine/thumbnails.php?album=7
http://www.2rosenthals.net/coppermine/thumbnails.php?album=favpics
http://www.2rosenthals.net/coppermine/thumbnails.php?album=lastcom&cat=0
http://www.2rosenthals.net/coppermine/thumbnails.php?album=lastcom&cat=10
http://www.2rosenthals.net/coppermine/thumbnails.php?album=lastcom&cat=13
http://www.2rosenthals.net/coppermine/thumbnails.php?album=lastcom&cat=-13
http://www.2rosenthals.net/coppermine/thumbnails.php?album=lastcom&cat=15
http://www.2rosenthals.net/coppermine/thumbnails.php?album=lastcom&cat=-15
http://www.2rosenthals.net/coppermine/thumbnails.php?album=lastcom&cat=-2
http://www.2rosenthals.net/coppermine/thumbnails.php?album=lastcom&cat=-24
http://www.2rosenthals.net/coppermine/thumbnails.php?album=lastcom&cat=5
http://www.2rosenthals.net/coppermine/thumbnails.php?album=lastcom&cat=-5
http://www.2rosenthals.net/coppermine/thumbnails.php?album=lastcom&cat=6
http://www.2rosenthals.net/coppermine/thumbnails.php?album=lastcom&cat=-6
http://www.2rosenthals.net/coppermine/thumbnails.php?album=lastcom&cat=7
http://www.2rosenthals.net/coppermine/thumbnails.php?album=lastcom&cat=8
http://www.2rosenthals.net/coppermine/thumbnails.php?album=lastcom&cat=9
http://www.2rosenthals.net/coppermine/thumbnails.php?album=lastup
http://www.2rosenthals.net/coppermine/thumbnails.php?album=lastup&cat=0
http://www.2rosenthals.net/coppermine/thumbnails.php?album=lastup&cat=10
http://www.2rosenthals.net/coppermine/thumbnails.php?album=lastup&cat=13
http://www.2rosenthals.net/coppermine/thumbnails.php?album=lastup&cat=14
http://www.2rosenthals.net/coppermine/thumbnails.php?album=lastup&cat=15
http://www.2rosenthals.net/coppermine/thumbnails.php?album=lastup&cat=-16
http://www.2rosenthals.net/coppermine/thumbnails.php?album=lastup&cat=17
http://www.2rosenthals.net/coppermine/thumbnails.php?album=lastup&cat=-18
http://www.2rosenthals.net/coppermine/thumbnails.php?album=lastup&cat=2
http://www.2rosenthals.net/coppermine/thumbnails.php?album=lastup&cat=-24
http://www.2rosenthals.net/coppermine/thumbnails.php?album=lastup&cat=4
http://www.2rosenthals.net/coppermine/thumbnails.php?album=lastup&cat=5
http://www.2rosenthals.net/coppermine/thumbnails.php?album=lastup&cat=-7
http://www.2rosenthals.net/coppermine/thumbnails.php?album=lastup&cat=8
http://www.2rosenthals.net/coppermine/thumbnails.php?album=lastup&cat=9
http://www.2rosenthals.net/coppermine/thumbnails.php?album=search&keywords=on&search=2001
http://www.2rosenthals.net/coppermine/thumbnails.php?album=search&keywords=on&search=3rd
http://www.2rosenthals.net/coppermine/thumbnails.php?album=search&keywords=on&search=ahoy
http://www.2rosenthals.net/coppermine/thumbnails.php?album=search&keywords=on&search=apron
http://www.2rosenthals.net/coppermine/thumbnails.php?album=search&keywords=on&search=art
http://www.2rosenthals.net/coppermine/thumbnails.php?album=search&keywords=on&search=awning
http://www.2rosenthals.net/coppermine/thumbnails.php?album=search&keywords=on&search=back
http://www.2rosenthals.net/coppermine/thumbnails.php?album=search&keywords=on&search=cadillac
http://www.2rosenthals.net/coppermine/thumbnails.php?album=search&keywords=on&search=cake
http://www.2rosenthals.net/coppermine/thumbnails.php?album=search&keywords=on&search=calvert
http://www.2rosenthals.net/coppermine/thumbnails.php?album=search&keywords=on&search=car
http://www.2rosenthals.net/coppermine/thumbnails.php?album=search&keywords=on&search=center
http://www.2rosenthals.net/coppermine/thumbnails.php?album=search&keywords=on&search=curriculum
http://www.2rosenthals.net/coppermine/thumbnails.php?album=search&keywords=on&search=dashboard
http://www.2rosenthals.net/coppermine/thumbnails.php?album=search&keywords=on&search=decklid
http://www.2rosenthals.net/coppermine/thumbnails.php?album=search&keywords=on&search=dog
http://www.2rosenthals.net/coppermine/thumbnails.php?album=search&keywords=on&search=door
http://www.2rosenthals.net/coppermine/thumbnails.php?album=search&keywords=on&search=drawing
http://www.2rosenthals.net/coppermine/thumbnails.php?album=search&keywords=on&search=dvd
http://www.2rosenthals.net/coppermine/thumbnails.php?album=search&keywords=on&search=fender
http://www.2rosenthals.net/coppermine/thumbnails.php?album=search&keywords=on&search=frame
http://www.2rosenthals.net/coppermine/thumbnails.php?album=search&keywords=on&search=gas
http://www.2rosenthals.net/coppermine/thumbnails.php?album=search&keywords=on&search=grade
http://www.2rosenthals.net/coppermine/thumbnails.php?album=search&keywords=on&search=grill
http://www.2rosenthals.net/coppermine/thumbnails.php?album=search&keywords=on&search=gutter
http://www.2rosenthals.net/coppermine/thumbnails.php?album=search&keywords=on&search=hills
http://www.2rosenthals.net/coppermine/thumbnails.php?album=search&keywords=on&search=indoors
http://www.2rosenthals.net/coppermine/thumbnails.php?album=search&keywords=on&search=inside
http://www.2rosenthals.net/coppermine/thumbnails.php?album=search&keywords=on&search=interior
http://www.2rosenthals.net/coppermine/thumbnails.php?album=search&keywords=on&search=issue
http://www.2rosenthals.net/coppermine/thumbnails.php?album=search&keywords=on&search=kitchen
http://www.2rosenthals.net/coppermine/thumbnails.php?album=search&keywords=on&search=left
http://www.2rosenthals.net/coppermine/thumbnails.php?album=search&keywords=on&search=lesson
http://www.2rosenthals.net/coppermine/thumbnails.php?album=search&keywords=on&search=lilac
http://www.2rosenthals.net/coppermine/thumbnails.php?album=search&keywords=on&search=luke
http://www.2rosenthals.net/coppermine/thumbnails.php?album=search&keywords=on&search=montage
http://www.2rosenthals.net/coppermine/thumbnails.php?album=search&keywords=on&search=moulding
http://www.2rosenthals.net/coppermine/thumbnails.php?album=search&keywords=on&search=new
http://www.2rosenthals.net/coppermine/thumbnails.php?album=search&keywords=on&search=oak
http://www.2rosenthals.net/coppermine/thumbnails.php?album=search&keywords=on&search=outside
http://www.2rosenthals.net/coppermine/thumbnails.php?album=search&keywords=on&search=panel
http://www.2rosenthals.net/coppermine/thumbnails.php?album=search&keywords=on&search=parts
http://www.2rosenthals.net/coppermine/thumbnails.php?album=search&keywords=on&search=passenger
http://www.2rosenthals.net/coppermine/thumbnails.php?album=search&keywords=on&search=present
http://www.2rosenthals.net/coppermine/thumbnails.php?album=search&keywords=on&search=profile
http://www.2rosenthals.net/coppermine/thumbnails.php?album=search&keywords=on&search=rachie
http://www.2rosenthals.net/coppermine/thumbnails.php?album=search&keywords=on&search=santa
http://www.2rosenthals.net/coppermine/thumbnails.php?album=search&keywords=on&search=site
http://www.2rosenthals.net/coppermine/thumbnails.php?album=search&keywords=on&search=spring
http://www.2rosenthals.net/coppermine/thumbnails.php?album=search&keywords=on&search=step
http://www.2rosenthals.net/coppermine/thumbnails.php?album=search&keywords=on&search=swing
http://www.2rosenthals.net/coppermine/thumbnails.php?album=search&keywords=on&search=tank
http://www.2rosenthals.net/coppermine/thumbnails.php?album=search&keywords=on&search=top
http://www.2rosenthals.net/coppermine/thumbnails.php?album=search&keywords=on&search=tree
http://www.2rosenthals.net/coppermine/thumbnails.php?album=search&keywords=on&search=tricycle
http://www.2rosenthals.net/coppermine/thumbnails.php?album=search&keywords=on&search=trim
http://www.2rosenthals.net/coppermine/thumbnails.php?album=search&keywords=on&search=windshield
http://www.2rosenthals.net/coppermine/thumbnails.php?album=search&keywords=on&search=wink
http://www.2rosenthals.net/coppermine/thumbnails.php?album=search&keywords=on&search=woodpile
http://www.2rosenthals.net/coppermine/thumbnails.php?album=search&keywords=on&search=yaya
http://www.2rosenthals.net/coppermine/thumbnails.php?album=search&keywords=on&search=york
http://www.2rosenthals.net/coppermine/thumbnails.php?album=topn
http://www.2rosenthals.net/coppermine/thumbnails.php?album=topn&cat=0
http://www.2rosenthals.net/coppermine/thumbnails.php?album=topn&cat=1
http://www.2rosenthals.net/coppermine/thumbnails.php?album=topn&cat=10
http://www.2rosenthals.net/coppermine/thumbnails.php?album=topn&cat=11
http://www.2rosenthals.net/coppermine/thumbnails.php?album=topn&cat=13
http://www.2rosenthals.net/coppermine/thumbnails.php?album=topn&cat=14
http://www.2rosenthals.net/coppermine/thumbnails.php?album=topn&cat=15
http://www.2rosenthals.net/coppermine/thumbnails.php?album=topn&cat=-15
http://www.2rosenthals.net/coppermine/thumbnails.php?album=topn&cat=16
http://www.2rosenthals.net/coppermine/thumbnails.php?album=topn&cat=-17
http://www.2rosenthals.net/coppermine/thumbnails.php?album=topn&cat=-18
http://www.2rosenthals.net/coppermine/thumbnails.php?album=topn&cat=2
http://www.2rosenthals.net/coppermine/thumbnails.php?album=topn&cat=-24
http://www.2rosenthals.net/coppermine/thumbnails.php?album=topn&cat=4
http://www.2rosenthals.net/coppermine/thumbnails.php?album=topn&cat=-5
http://www.2rosenthals.net/coppermine/thumbnails.php?album=topn&cat=6
http://www.2rosenthals.net/coppermine/thumbnails.php?album=topn&cat=-6
http://www.2rosenthals.net/coppermine/thumbnails.php?album=topn&cat=-7
http://www.2rosenthals.net/coppermine/thumbnails.php?album=topn&cat=8
http://www.2rosenthals.net/coppermine/thumbnails.php?album=toprated&cat=0
http://www.2rosenthals.net/coppermine/thumbnails.php?album=toprated&cat=1
http://www.2rosenthals.net/coppermine/thumbnails.php?album=toprated&cat=10
http://www.2rosenthals.net/coppermine/thumbnails.php?album=toprated&cat=13
http://www.2rosenthals.net/coppermine/thumbnails.php?album=toprated&cat=-13
http://www.2rosenthals.net/coppermine/thumbnails.php?album=toprated&cat=14
http://www.2rosenthals.net/coppermine/thumbnails.php?album=toprated&cat=16
http://www.2rosenthals.net/coppermine/thumbnails.php?album=toprated&cat=17
http://www.2rosenthals.net/coppermine/thumbnails.php?album=toprated&cat=-17
http://www.2rosenthals.net/coppermine/thumbnails.php?album=toprated&cat=-24
http://www.2rosenthals.net/coppermine/thumbnails.php?album=toprated&cat=3
http://www.2rosenthals.net/coppermine/thumbnails.php?album=toprated&cat=-5
http://www.2rosenthals.net/coppermine/thumbnails.php?album=toprated&cat=6
http://www.2rosenthals.net/coppermine/thumbnails.php?album=toprated&cat=7
http://www.2rosenthals.net/coppermine/thumbnails.php?album=toprated&cat=-7
http://www.2rosenthals.netlogin.php?force_login=1&referer=displayimage.php%3Falbum%3D15%26pid%3D237
http://www.2rosenthals.netlogin.php?force_login=1&referer=displayimage.php%3Falbum%3D15%26pid%3D315
http://www.2rosenthals.netlogin.php?force_login=1&referer=displayimage.php%3Falbum%3D18%26pid%3D189
http://www.2rosenthals.netlogin.php?force_login=1&referer=displayimage.php%3Falbum%3D18%26pid%3D349
http://www.2rosenthals.netlogin.php?force_login=1&referer=displayimage.php%3Falbum%3D7%26pid%3D564
http://www.2rosenthals.net/robots.txt
http://www.2rosenthals.net/wordpress
http://www.2rosenthals.net/wordpress/
http://www.2rosenthals.net/wordpress/abduction-save-as-image-mod-for-seamonkey-576/
http://www.2rosenthals.net/wordpress/abduction-save-as-image-mod-for-seamonkey-576/?action=lostpassword&instance=1
http://www.2rosenthals.net/wordpress/abduction-save-as-image-mod-for-seamonkey-576/?action=register&instance=1
http://www.2rosenthals.net/wordpress/abduction-save-as-image-mod-for-seamonkey-576/comment-page-1/
http://www.2rosenthals.net/wordpress/abduction-save-as-image-mod-for-seamonkey-576/feed/
http://www.2rosenthals.net/wordpress/abduction-save-as-image-mod-for-seamonkey-576/?format=pdf
http://www.2rosenthals.net/wordpress/?action=login&instance=1
http://www.2rosenthals.net/wordpress/?action=lostpassword&instance=1
http://www.2rosenthals.net/wordpress/betterprivacy-mod-for-seamonkey-532/
http://www.2rosenthals.net/wordpress/betterprivacy-mod-for-seamonkey-532/?format=pdf
http://www.2rosenthals.net/wordpress/cloud-adoption-brings-unexpected-costs-kpmg-survey-says-583/?action=register&instance=1
http://www.2rosenthals.net/wordpress/cloud-adoption-brings-unexpected-costs-kpmg-survey-says-583/feed/
http://www.2rosenthals.net/wordpress/cloud-adoption-brings-unexpected-costs-kpmg-survey-says-583/?format=pdf
http://www.2rosenthals.net/wordpress/comments/feed/
http://www.2rosenthals.net/wordpress/configuring-squid-proxy-on-os2-path-adjustments-399/
http://www.2rosenthals.net/wordpress/contact/
http://www.2rosenthals.net/wordpress/core/core-images/backgrounds/low_contrast_linen.png
http://www.2rosenthals.net/wordpress/crts-vs-lcds-in-20112012-388/
http://www.2rosenthals.net/wordpress/date/2013/02/
http://www.2rosenthals.net/wordpress/egad-why-do-people-do-their-own-web-development-384/
http://www.2rosenthals.net/wordpress/facebook-threatens-to-shut-down-conservative-site-fox-news-commentary-todd-starnes-582/
http://www.2rosenthals.net/wordpress/facebook-threatens-to-shut-down-conservative-site-fox-news-commentary-todd-starnes-582/?action=register&instance=1
http://www.2rosenthals.net/wordpress/facebook-threatens-to-shut-down-conservative-site-fox-news-commentary-todd-starnes-582/feed/
http://www.2rosenthals.net/wordpress/feed/
http://www.2rosenthals.net/wordpress/file/?action=register&instance=1
http://www.2rosenthals.net/wordpress/file/feed/
http://www.2rosenthals.net/wordpress/firesheep-not-on-a-hautspot-network-123/
http://www.2rosenthals.net/wordpress/group-chat-showdown-which-instant-messaging-service-is-best-for-your-business-pcworld-523/
http://www.2rosenthals.net/wordpress/help/
http://www.2rosenthals.net/wordpress/help/about/
http://www.2rosenthals.net/wordpress/help/about/?action=lostpassword&instance=1
http://www.2rosenthals.net/wordpress/help/about/?action=register&instance=1
http://www.2rosenthals.net/wordpress/help/?action=lostpassword&instance=1
http://www.2rosenthals.net/wordpress/index.php?=PHPE9568F34-D428-11d2-A769-00AA001ACF42
http://www.2rosenthals.net/wordpress/index.php?=PHPE9568F35-D428-11d2-A769-00AA001ACF42
http://www.2rosenthals.net/wordpress/installing-windows-server-2008-r2-x64-on-the-hp-proliant-dl380-g4-108/
http://www.2rosenthals.net/wordpress/login/
http://www.2rosenthals.net/wordpress/login/?action=logout&_wpnonce=64c145177b
http://www.2rosenthals.net/wordpress/login/?action=profile
http://www.2rosenthals.net/wordpress/login/?loggedout=true
http://www.2rosenthals.net/wordpress/login/?redirect_to=http%3A%2F%2Fwww.2rosenthals.net%2Fwordpress%2Fcloud-adoption-brings-unexpected-costs-kpmg-survey-says-583%2F
http://www.2rosenthals.net/wordpress/login/?redirect_to=http%3A%2F%2Fwww.2rosenthals.net%2Fwordpress%2Ffile%2F
http://www.2rosenthals.net/wordpress/login/?redirect_to=http%3A%2F%2Fwww.2rosenthals.net%2Fwordpress%2Fremoving-the-wp-post-to-pdf-icon-in-wptouch-569%2F
http://www.2rosenthals.net/wordpress/login/?redirect_to=http%3A%2F%2Fwww.2rosenthals.net%2Fwordpress%2Fwp-admin%2Fplugin-install.php&reauth=1
http://www.2rosenthals.net/wordpress/login/?redirect_to=http%3A%2F%2Fwww.2rosenthals.net%2Fwordpress%2Fwp-admin%2Fupdate-core.php&reauth=1
http://www.2rosenthals.net/wordpress/mantis-email-fun-part-2-362/
http://www.2rosenthals.net/wordpress/mass-editing-users-in-wordpress-528/
http://www.2rosenthals.net/wordpress/more-cloud-disasters-499/
http://www.2rosenthals.net/wordpress/more-cloud-disasters-499/?format=pdf
http://www.2rosenthals.net/wordpress/mozilla-hardware-acceleration-and-vnc-sessions-186/
http://www.2rosenthals.net/wordpress/new-upgrade-yet-again-492/
http://www.2rosenthals.net/wordpress/on-the-ungrateful-nature-of-users-and-the-oss-development-community-438/
http://www.2rosenthals.net/wordpress/?p=569
http://www.2rosenthals.net/wordpress/?p=582
http://www.2rosenthals.net/wordpress/?p=584
http://www.2rosenthals.net/wordpress/?p=584&preview=true
http://www.2rosenthals.net/wordpress/?p=596&preview=true
http://www.2rosenthals.net/wordpress/page/2/
http://www.2rosenthals.net/wordpress/ramdom-thoughts-on-the-2011-and-beyond-firefox-release-schedule-223/
http://www.2rosenthals.net/wordpress/random-thoughts-on-thunderbirds-current-state-495/
http://www.2rosenthals.net/wordpress/removing-the-wp-post-to-pdf-icon-in-wptouch-569/?action=lostpassword&instance=1
http://www.2rosenthals.net/wordpress/removing-the-wp-post-to-pdf-icon-in-wptouch-569/feed/
http://www.2rosenthals.net/wordpress/removing-the-wp-post-to-pdf-icon-in-wptouch-569/?format=pdf
http://www.2rosenthals.net/wordpress/rsync-error-5-may-be-exactly-what-it-says-165/
http://www.2rosenthals.net/wordpress/running-rconsolej-on-os2-483/
http://www.2rosenthals.net/wordpress/tag/64-bit/
http://www.2rosenthals.net/wordpress/tag/apache/
http://www.2rosenthals.net/wordpress/tag/arena/
http://www.2rosenthals.net/wordpress/tag/buzzword/
http://www.2rosenthals.net/wordpress/tag/chip/
http://www.2rosenthals.net/wordpress/tag/chrome/
http://www.2rosenthals.net/wordpress/tag/cloud/
http://www.2rosenthals.net/wordpress/tag/cloud/?action=register&instance=1
http://www.2rosenthals.net/wordpress/tag/conf/
http://www.2rosenthals.net/wordpress/tag/consult/
http://www.2rosenthals.net/wordpress/tag/crt/
http://www.2rosenthals.net/wordpress/tag/ddos/
http://www.2rosenthals.net/wordpress/tag/directory/
http://www.2rosenthals.net/wordpress/tag/ecomstation/
http://www.2rosenthals.net/wordpress/tag/ecs-2/
http://www.2rosenthals.net/wordpress/tag/extended-attributes/
http://www.2rosenthals.net/wordpress/tag/facebook/
http://www.2rosenthals.net/wordpress/tag/facebook/?action=register&instance=1
http://www.2rosenthals.net/wordpress/tag/files/
http://www.2rosenthals.net/wordpress/tag/files/?action=lostpassword&instance=1
http://www.2rosenthals.net/wordpress/tag/files/?action=register&instance=1
http://www.2rosenthals.net/wordpress/tag/files/feed/
http://www.2rosenthals.net/wordpress/tag/files/three-good-ways-to-thwart-tracking-attempts-on-the-net-251/
http://www.2rosenthals.net/wordpress/tag/filesystem/
http://www.2rosenthals.net/wordpress/tag/firefox/?action=register&instance=1
http://www.2rosenthals.net/wordpress/tag/fix/
http://www.2rosenthals.net/wordpress/tag/fix/?action=register&instance=1
http://www.2rosenthals.net/wordpress/tag/foss/
http://www.2rosenthals.net/wordpress/tag/fragmentation/
http://www.2rosenthals.net/wordpress/tag/green/
http://www.2rosenthals.net/wordpress/tag/head-parking/
http://www.2rosenthals.net/wordpress/tag/heat/
http://www.2rosenthals.net/wordpress/tag/hosted/
http://www.2rosenthals.net/wordpress/tag/how-to/
http://www.2rosenthals.net/wordpress/tag/humor/
http://www.2rosenthals.net/wordpress/tag/hype/
http://www.2rosenthals.net/wordpress/tag/hype/?action=register&instance=1
http://www.2rosenthals.net/wordpress/tag/hype/feed/
http://www.2rosenthals.net/wordpress/tag/images/
http://www.2rosenthals.net/wordpress/tag/images/?action=register&instance=1
http://www.2rosenthals.net/wordpress/tag/install/
http://www.2rosenthals.net/wordpress/tag/internet-2/?action=lostpassword&instance=1
http://www.2rosenthals.net/wordpress/tag/internet-2/?action=register&instance=1
http://www.2rosenthals.net/wordpress/tag/internet-2/feed/
http://www.2rosenthals.net/wordpress/tag/iproute2/
http://www.2rosenthals.net/wordpress/tag/jabber/
http://www.2rosenthals.net/wordpress/tag/jfs/
http://www.2rosenthals.net/wordpress/tag/lcd/
http://www.2rosenthals.net/wordpress/tag/linux/
http://www.2rosenthals.net/wordpress/tag/management/
http://www.2rosenthals.net/wordpress/tag/mantis-2/
http://www.2rosenthals.net/wordpress/tag/mozilla/
http://www.2rosenthals.net/wordpress/tag/mozilla/?action=lostpassword&instance=1
http://www.2rosenthals.net/wordpress/tag/mozilla/feed/
http://www.2rosenthals.net/wordpress/tag/msf/
http://www.2rosenthals.net/wordpress/tag/myspace/
http://www.2rosenthals.net/wordpress/tag/mysql-2/
http://www.2rosenthals.net/wordpress/tag/netscape/
http://www.2rosenthals.net/wordpress/tag/noise/
http://www.2rosenthals.net/wordpress/tag/nvidia/
http://www.2rosenthals.net/wordpress/tag/oss/
http://www.2rosenthals.net/wordpress/tag/perl/
http://www.2rosenthals.net/wordpress/tag/php-2/
http://www.2rosenthals.net/wordpress/tag/power-management/
http://www.2rosenthals.net/wordpress/tag/privacy/?action=lostpassword&instance=1
http://www.2rosenthals.net/wordpress/tag/proliant/
http://www.2rosenthals.net/wordpress/tag/quality/
http://www.2rosenthals.net/wordpress/tag/release-cycle/
http://www.2rosenthals.net/wordpress/tag/repair/
http://www.2rosenthals.net/wordpress/tag/rim/
http://www.2rosenthals.net/wordpress/tag/routing-2/
http://www.2rosenthals.net/wordpress/tag/script/
http://www.2rosenthals.net/wordpress/tag/seamonkey/
http://www.2rosenthals.net/wordpress/tag/seamonkey/?action=lostpassword&instance=1
http://www.2rosenthals.net/wordpress/tag/seamonkey/?action=register&instance=1
http://www.2rosenthals.net/wordpress/tag/security/
http://www.2rosenthals.net/wordpress/tag/server-2008/
http://www.2rosenthals.net/wordpress/tag/site/
http://www.2rosenthals.net/wordpress/tag/social-networking/feed/
http://www.2rosenthals.net/wordpress/tag/squid/
http://www.2rosenthals.net/wordpress/tag/symlink/
http://www.2rosenthals.net/wordpress/tag/technology/
http://www.2rosenthals.net/wordpress/tag/thinkpad/
http://www.2rosenthals.net/wordpress/tag/time-wasters/
http://www.2rosenthals.net/wordpress/tag/troubleshooting/
http://www.2rosenthals.net/wordpress/tag/tuning/
http://www.2rosenthals.net/wordpress/tag/twitter/
http://www.2rosenthals.net/wordpress/tag/update/
http://www.2rosenthals.net/wordpress/tag/upgrade/
http://www.2rosenthals.net/wordpress/tag/vnc/
http://www.2rosenthals.net/wordpress/tag/windows/
http://www.2rosenthals.net/wordpress/tag/wordpress/
http://www.2rosenthals.net/wordpress/tag/workplace-shell/
http://www.2rosenthals.net/wordpress/tag/x64/
http://www.2rosenthals.net/wordpress/tag/xmpp/
http://www.2rosenthals.net/wordpress/the-strangest-laptop-repair-ive-ever-done-127/
http://www.2rosenthals.net/wordpress/the-strangest-laptop-repair-ive-ever-done-127/comment-page-1/
http://www.2rosenthals.net/wordpress/the-telltale-hard-drive-173/
http://www.2rosenthals.net/wordpress/updating-firefox-on-linux-the-dreaded-couldnt-load-xpcom-190/
http://www.2rosenthals.net/wordpress/warpstock-2012-portland-oregon-usa-488/
http://www.2rosenthals.net/wordpress/welcome/
http://www.2rosenthals.net/wordpress/welcome/?action=register&instance=1
http://www.2rosenthals.net/wordpress/welcome/?page_id=44
http://www.2rosenthals.net/wordpress/why-i-still-use-os2-ecomstation-436/
http://www.2rosenthals.net/wordpress/with-the-coming-of-windows-8-there-has-never-been-a-better-time-to-switch-platforms-498/
http://www.2rosenthals.net/wordpress/workaround-multi-column-footers-in-openoffice-org-writer-540/
http://www.2rosenthals.net/wordpress/wp-admin/
http://www.2rosenthals.net/wordpress/wp-admin/admin-ajax.php
http://www.2rosenthals.net/wordpress/wp-admin/admin-ajax.php?action=ajax-tag-search&tax=post_tag&q=clie
http://www.2rosenthals.net/wordpress/wp-admin/admin-ajax.php?action=ajax-tag-search&tax=post_tag&q=client
http://www.2rosenthals.net/wordpress/wp-admin/admin-ajax.php?action=ajax-tag-search&tax=post_tag&q=pr
http://www.2rosenthals.net/wordpress/wp-admin/admin-ajax.php?action=ajax-tag-search&tax=post_tag&q=pract
http://www.2rosenthals.net/wordpress/wp-admin/admin-ajax.php?action=ajax-tag-search&tax=post_tag&q=practi
http://www.2rosenthals.net/wordpress/wp-admin/admin-ajax.php?action=ajax-tag-search&tax=post_tag&q=practice
http://www.2rosenthals.net/wordpress/wp-admin/admin-ajax.php?action=ajax-tag-search&tax=post_tag&q=przc
http://www.2rosenthals.net/wordpress/wp-admin/admin-ajax.php?action=dashboard-widgets&widget=dashboard_incoming_links
http://www.2rosenthals.net/wordpress/wp-admin/admin-ajax.php?action=dashboard-widgets&widget=dashboard_plugins
http://www.2rosenthals.net/wordpress/wp-admin/admin-ajax.php?action=dashboard-widgets&widget=dashboard_primary
http://www.2rosenthals.net/wordpress/wp-admin/admin-ajax.php?action=dashboard-widgets&widget=dashboard_secondary
http://www.2rosenthals.net/wordpress/wp-admin/admin-ajax.php?action=oembed-cache&post=569
http://www.2rosenthals.net/wordpress/wp-admin/admin.php?page=ckeditor_advanced_options
http://www.2rosenthals.net/wordpress/wp-admin/admin.php?page=ckeditor_basic_options
http://www.2rosenthals.net/wordpress/wp-admin/admin.php?page=ckeditor_settings
http://www.2rosenthals.net/wordpress/wp-admin/admin.php?page=stats&noheader=1
http://www.2rosenthals.net/wordpress/wp-admin/css/colors-fresh.min.css?ver=3.5.1
http://www.2rosenthals.net/wordpress/wp-admin/edit.php
http://www.2rosenthals.net/wordpress/wp-admin/images/arrows.png
http://www.2rosenthals.net/wordpress/wp-admin/images/bubble_bg.gif
http://www.2rosenthals.net/wordpress/wp-admin/images/button-grad.png
http://www.2rosenthals.net/wordpress/wp-admin/images/comment-grey-bubble.png
http://www.2rosenthals.net/wordpress/wp-admin/images/date-button.gif
http://www.2rosenthals.net/wordpress/wp-admin/images/icons32-2x.png?ver=20121105
http://www.2rosenthals.net/wordpress/wp-admin/images/icons32.png?ver=20111206
http://www.2rosenthals.net/wordpress/wp-admin/images/icons32.png?ver=20121105
http://www.2rosenthals.net/wordpress/wp-admin/images/list.png
http://www.2rosenthals.net/wordpress/wp-admin/images/media-button-2x.png
http://www.2rosenthals.net/wordpress/wp-admin/images/media-button.png
http://www.2rosenthals.net/wordpress/wp-admin/images/menu.png?ver=20120201
http://www.2rosenthals.net/wordpress/wp-admin/images/menu.png?ver=20121105
http://www.2rosenthals.net/wordpress/wp-admin/images/menu-shadow.png
http://www.2rosenthals.net/wordpress/wp-admin/images/press-this.png?v=20120502
http://www.2rosenthals.net/wordpress/wp-admin/images/resize.gif
http://www.2rosenthals.net/wordpress/wp-admin/images/sort.gif
http://www.2rosenthals.net/wordpress/wp-admin/images/stars.png?ver=20121108
http://www.2rosenthals.net/wordpress/wp-admin/images/white-grad-active.png
http://www.2rosenthals.net/wordpress/wp-admin/images/white-grad.png
http://www.2rosenthals.net/wordpress/wp-admin/images/wordpress-logo-2x.png?ver=20120412
http://www.2rosenthals.net/wordpress/wp-admin/images/wpspin_light.gif
http://www.2rosenthals.net/wordpress/wp-admin/images/xit.gif
http://www.2rosenthals.net/wordpress/wp-admin/index.php
http://www.2rosenthals.net/wordpress/wp-admin/js/password-strength-meter.min.js?ver=3.5.1
http://www.2rosenthals.net/wordpress/wp-admin/js/user-profile.min.js?ver=3.5.1
http://www.2rosenthals.net/wordpress/wp-admin/load-scripts.php?c=0&load=admin-bar,hoverIntent,common,jquery-color,thickbox,plugin-install,jquery-ui-widget,jquery-ui-position,wp-pointer,jquery-ui-core,jquery-ui-datepicker&ver=3.4.2
http://www.2rosenthals.net/wordpress/wp-admin/load-scripts.php?c=0&load=admin-bar,hoverIntent,common,jquery-color,thickbox,theme,customize-base,customize-loader,jquery-ui-widget,jquery-ui-position,wp-pointer&ver=3.4.2
http://www.2rosenthals.net/wordpress/wp-admin/load-scripts.php?c=0&load=admin-bar,hoverIntent,common,jquery-color,thickbox,wp-ajax-response,wp-lists,quicktags,jquery-query,admin-comments,jquery-ui-core,jquery-ui-widget,jquery-ui-mouse,jquery-ui-sortable,postbox,dashboard,plugin-install,media-upload,jquery-ui-position,wp-pointer,editor&ver=3.4.2
http://www.2rosenthals.net/wordpress/wp-admin/load-scripts.php?c=0&load[]=admin-bar,hoverIntent,common,thickbox,plugin-install,jquery-ui-widget,jquery-ui-position,wp-pointer&ver=3.5
http://www.2rosenthals.net/wordpress/wp-admin/load-scripts.php?c=0&load=hoverIntent,common,jquery-color,thickbox,jquery-ui-widget,jquery-ui-position,wp-pointer&ver=3.4.2
http://www.2rosenthals.net/wordpress/wp-admin/load-scripts.php?c=0&load=jquery,utils,json2&ver=3.4.2
http://www.2rosenthals.net/wordpress/wp-admin/load-scripts.php?c=0&load=jquery,utils&ver=3.4.2
http://www.2rosenthals.net/wordpress/wp-admin/load-scripts.php?c=1&load%5B%5D=admin-bar,hoverIntent,common,jquery-ui-core,jquery-ui-widget,jquery-ui-mouse,jquery-ui-sortable,postbox,thickbox,jquery-ui-posit&load%5B%5D=ion,wp-pointer&ver=3.5.1
http://www.2rosenthals.net/wordpress/wp-admin/load-scripts.php?c=1&load%5B%5D=admin-bar,hoverIntent,common,thickbox,jquery-ui-widget,jquery-ui-position,wp-pointer&ver=3.5.1
http://www.2rosenthals.net/wordpress/wp-admin/load-scripts.php?c=1&load%5B%5D=admin-bar,hoverIntent,common,thickbox,plugin-install,jquery-ui-widget,jquery-ui-position,wp-pointer,jquery-ui-core,jquery-ui-dat&load%5B%5D=epicker&ver=3.5.1
http://www.2rosenthals.net/wordpress/wp-admin/load-scripts.php?c=1&load%5B%5D=admin-bar,hoverIntent,common,thickbox,plugin-install,jquery-ui-widget,jquery-ui-position,wp-pointer&ver=3.5.1
http://www.2rosenthals.net/wordpress/wp-admin/load-scripts.php?c=1&load%5B%5D=admin-bar,hoverIntent,common,thickbox,schedule,wp-ajax-response,autosave,jquery-color,wp-lists,quicktags,jquery-query,admin-comm&load%5B%5D=ents,suggest,jquery-ui-core,jquery-ui-widget,jquery-ui-mouse,jquery-ui-sortable,postbox,post,underscore,shortcode,backbone,media&load%5B%5D=-models,wp-plupload,media-views,media-editor,jquery-ui-position,wp-pointer,editor&ver=3.5.1
http://www.2rosenthals.net/wordpress/wp-admin/load-scripts.php?c=1&load%5B%5D=admin-bar,hoverIntent,common,thickbox,schedule,wp-ajax-response,autosave,jquery-color,wp-lists,quicktags,jquery-query,admin-comm&load%5B%5D=ents,suggest,jquery-ui-core,jquery-ui-widget,jquery-ui-mouse,jquery-ui-sortable,postbox,post,underscore,shortcode,backbone,media&load%5B%5D=-models,wp-plupload,media-views,media-editor,jquery-ui-position,wp-pointer,word-count,editor,jquery-ui-resizable,jquery-ui-dragg&load%5B%5D=able,jquery-ui-button,jquery-ui-dialog,wpdialogs,wplink,wpdialogs-popup,wp-fullscreen,media-upload&ver=3.5.1
http://www.2rosenthals.net/wordpress/wp-admin/load-scripts.php?c=1&load%5B%5D=admin-bar,hoverIntent,common,thickbox,schedule,wp-ajax-response,autosave,suggest,jquery-color,wp-lists,jquery-ui-core,jquery-ui-&load%5B%5D=widget,jquery-ui-mouse,jquery-ui-sortable,postbox,post,underscore,shortcode,backbone,media-models,wp-plupload,media-views,media-&load%5B%5D=editor,jquery-ui-position,wp-pointer,word-count,editor,quicktags,jquery-ui-resizable,jquery-ui-draggable,jquery-ui-button,jquery&load%5B%5D=-ui-dialog,wpdialogs,wplink,wpdialogs-popup,wp-fullscreen,media-upload&ver=3.5.1
http://www.2rosenthals.net/wordpress/wp-admin/load-scripts.php?c=1&load%5B%5D=admin-bar,hoverIntent,common,thickbox,suggest,inline-edit-post,jquery-ui-widget,jquery-ui-position,wp-pointer&ver=3.5.1
http://www.2rosenthals.net/wordpress/wp-admin/load-scripts.php?c=1&load%5B%5D=admin-bar,hoverIntent,common,thickbox,wp-ajax-response,jquery-color,wp-lists,quicktags,jquery-query,admin-comments,jquery-ui-cor&load%5B%5D=e,jquery-ui-widget,jquery-ui-mouse,jquery-ui-sortable,postbox,dashboard,customize-base,customize-loader,plugin-install,underscor&load%5B%5D=e,shortcode,media-upload,jquery-ui-position,wp-pointer,editor,backbone,media-models,plupload,plupload-html5,plupload-flash,plupl&load%5B%5D=oad-silverlight,plupload-html4,wp-plupload,media-views,media-editor&ver=3.5.1
http://www.2rosenthals.net/wordpress/wp-admin/load-scripts.php?c=1&load%5B%5D=hoverIntent,common,thickbox,jquery-ui-widget,jquery-ui-position,wp-pointer&ver=3.5.1
http://www.2rosenthals.net/wordpress/wp-admin/load-scripts.php?c=1&load%5B%5D=jquery,utils,json2&ver=3.5.1
http://www.2rosenthals.net/wordpress/wp-admin/load-scripts.php?c=1&load%5B%5D=jquery,utils,plupload,plupload-html5,plupload-flash,plupload-silverlight,plupload-html4,json2&ver=3.5.1
http://www.2rosenthals.net/wordpress/wp-admin/load-scripts.php?c=1&load%5B%5D=jquery,utils&ver=3.5.1
http://www.2rosenthals.net/wordpress/wp-admin/load-scripts.php?c=1&load[]=admin-bar,hoverIntent,common,thickbox,plugin-install,jquery-ui-widget,jquery-ui-position,wp-pointer,jquery-ui-core,jquery-ui-dat&load[]=epicker&ver=3.5
http://www.2rosenthals.net/wordpress/wp-admin/load-scripts.php?c=1&load[]=admin-bar,hoverIntent,common,thickbox,plugin-install,jquery-ui-widget,jquery-ui-position,wp-pointer&ver=3.5
http://www.2rosenthals.net/wordpress/wp-admin/load-scripts.php?c=1&load[]=admin-bar,hoverIntent,common,thickbox,schedule,wp-ajax-response,autosave,jquery-color,wp-lists,quicktags,jquery-query,admin-comm&load[]=ents,suggest,jquery-ui-core,jquery-ui-widget,jquery-ui-mouse,jquery-ui-sortable,postbox,post,underscore,shortcode,backbone,media&load[]=-models,wp-plupload,media-views,media-editor,jquery-ui-position,wp-pointer,editor&ver=3.5
http://www.2rosenthals.net/wordpress/wp-admin/load-scripts.php?c=1&load[]=admin-bar,hoverIntent,common,thickbox,schedule,wp-ajax-response,autosave,suggest,jquery-color,wp-lists,jquery-ui-core,jquery-ui-&load[]=widget,jquery-ui-mouse,jquery-ui-sortable,postbox,post,underscore,shortcode,backbone,media-models,wp-plupload,media-views,media-&load[]=editor,jquery-ui-position,wp-pointer,editor&ver=3.5
http://www.2rosenthals.net/wordpress/wp-admin/load-scripts.php?c=1&load[]=admin-bar,hoverIntent,common,thickbox,suggest,inline-edit-post,jquery-ui-widget,jquery-ui-position,wp-pointer&ver=3.5
http://www.2rosenthals.net/wordpress/wp-admin/load-scripts.php?c=1&load[]=admin-bar,hoverIntent,common,thickbox,wp-ajax-response,jquery-color,wp-lists,quicktags,jquery-query,admin-comments,jquery-ui-cor&load[]=e,jquery-ui-widget,jquery-ui-mouse,jquery-ui-sortable,postbox,dashboard,customize-base,customize-loader,plugin-install,underscor&load[]=e,shortcode,media-upload,jquery-ui-position,wp-pointer,editor,backbone,media-models,plupload,plupload-html5,plupload-flash,plupl&load[]=oad-silverlight,plupload-html4,wp-plupload,media-views,media-editor&ver=3.5
http://www.2rosenthals.net/wordpress/wp-admin/load-scripts.php?c=1&load[]=hoverIntent,common,thickbox,jquery-ui-widget,jquery-ui-position,wp-pointer&ver=3.5
http://www.2rosenthals.net/wordpress/wp-admin/load-scripts.php?c=1&load[]=jquery-ui-core,jquery-ui-widget,jquery-ui-mouse,jquery-ui-resizable,jquery-ui-draggable,jquery-ui-button,jquery-ui-position,jque&load[]=ry-ui-dialog&ver=3.5
http://www.2rosenthals.net/wordpress/wp-admin/load-scripts.php?c=1&load[]=jquery,utils,json2&ver=3.5
http://www.2rosenthals.net/wordpress/wp-admin/load-scripts.php?c=1&load[]=jquery,utils&ver=3.5
http://www.2rosenthals.net/wordpress/wp-admin/load-scripts.php?c=1&load[]=jquery&ver=3.5
http://www.2rosenthals.net/wordpress/wp-admin/load-styles.php?c=0&dir=ltr&load=admin-bar,wp-admin,buttons,wp-pointer&ver=3.5
http://www.2rosenthals.net/wordpress/wp-admin/load-styles.php?c=0&dir=ltr&load=admin-bar,wp-admin,wp-pointer&ver=3.4.2
http://www.2rosenthals.net/wordpress/wp-admin/load-styles.php?c=0&dir=ltr&load=wp-admin,wp-pointer&ver=3.4.2
http://www.2rosenthals.net/wordpress/wp-admin/load-styles.php?c=1&dir=ltr&load=admin-bar,buttons,media-views,wp-admin,wp-pointer&ver=3.5.1
http://www.2rosenthals.net/wordpress/wp-admin/load-styles.php?c=1&dir=ltr&load=admin-bar,wp-admin,buttons,wp-pointer&ver=3.5
http://www.2rosenthals.net/wordpress/wp-admin/load-styles.php?c=1&dir=ltr&load=admin-bar,wp-admin,buttons,wp-pointer&ver=3.5.1
http://www.2rosenthals.net/wordpress/wp-admin/load-styles.php?c=1&dir=ltr&load=media-views&ver=3.5.1
http://www.2rosenthals.net/wordpress/wp-admin/load-styles.php?c=1&dir=ltr&load=wp-admin,buttons,wp-pointer&ver=3.5.1
http://www.2rosenthals.net/wordpress/wp-admin/load-styles.php?c=1&dir=ltr&load=wp-jquery-ui-dialog&ver=3.5.1
http://www.2rosenthals.net/wordpress/wp-admin/options-general.php
http://www.2rosenthals.net/wordpress/wp-admin/options-general.php?page=syntax-highlighter-compress.php
http://www.2rosenthals.net/wordpress/wp-admin/options-general.php?page=wp-post-to-pdf/wp-post-to-pdf.php
http://www.2rosenthals.net/wordpress/wp-admin/options.php
http://www.2rosenthals.net/wordpress/wp-admin/plugin-editor.php?file=wptouch%2Fadmin-css%2Fbnc-compressed-global.css&plugin=wptouch%2Fwptouch.php
http://www.2rosenthals.net/wordpress/wp-admin/plugin-editor.php?file=wptouch%2Fadmin-css%2Fbnc-global.css&plugin=wptouch%2Fadmin-css%2Fbnc-compressed-global.css
http://www.2rosenthals.net/wordpress/wp-admin/plugin-install.php
http://www.2rosenthals.net/wordpress/wp-admin/plugin-install.php?tab=plugin-information&plugin=print-me&
http://www.2rosenthals.net/wordpress/wp-admin/plugin-install.php?tab=search&s=wp+post+to+pdf&plugin-search-input=Search+Plugins
http://www.2rosenthals.net/wordpress/wp-admin/plugin-install.php?tab=search&type=term&s=email&plugin-search-input=Search+Plugins
http://www.2rosenthals.net/wordpress/wp-admin/plugins.php
http://www.2rosenthals.net/wordpress/wp-admin/plugins.php?action=activate&plugin=ckeditor-for-wordpress%2Fckeditor_wordpress.php&plugin_status=all&paged=1&s&_wpnonce=b0b02fa219
http://www.2rosenthals.net/wordpress/wp-admin/plugins.php?action=activate&plugin=wp-post-to-pdf%2Fwp-post-to-pdf.php&plugin_status=all&paged=1&s&_wpnonce=4cc7ee04d0
http://www.2rosenthals.net/wordpress/wp-admin/plugins.php?action=activate&plugin=wp-post-to-pdf%2Fwp-post-to-pdf.php&_wpnonce=4cc7ee04d0
http://www.2rosenthals.net/wordpress/wp-admin/plugins.php?action=deactivate&plugin=ckeditor-for-wordpress%2Fckeditor_wordpress.php&plugin_status=all&paged=1&s&_wpnonce=05d0ba2ffb
http://www.2rosenthals.net/wordpress/wp-admin/plugins.php?action=deactivate&plugin=wp-post-to-pdf%2Fwp-post-to-pdf.php&plugin_status=all&paged=1&s&_wpnonce=dce3298d87
http://www.2rosenthals.net/wordpress/wp-admin/plugins.php?activate=true&plugin_status=all&paged=1&s=
http://www.2rosenthals.net/wordpress/wp-admin/plugins.php?deactivate=true&plugin_status=all&paged=1&s=
http://www.2rosenthals.net/wordpress/wp-admin/post-new.php
http://www.2rosenthals.net/wordpress/wp-admin/post.php
http://www.2rosenthals.net/wordpress/wp-admin/post.php?post=44&action=edit
http://www.2rosenthals.net/wordpress/wp-admin/post.php?post=584&action=edit
http://www.2rosenthals.net/wordpress/wp-admin/post.php?post=584&action=edit&message=10
http://www.2rosenthals.net/wordpress/wp-admin/post.php?post=584&action=edit&message=6
http://www.2rosenthals.net/wordpress/wp-admin/post.php?post=596&action=edit
http://www.2rosenthals.net/wordpress/wp-admin/post.php?post=596&action=edit&message=10
http://www.2rosenthals.net/wordpress/wp-admin/theme-editor.php
http://www.2rosenthals.net/wordpress/wp-admin/theme-editor.php?file=page.php&theme=lightword-2.0.0.6
http://www.2rosenthals.net/wordpress/wp-admin/theme-editor.php?file=yarpp-template-example.php&theme=lightword-child
http://www.2rosenthals.net/wordpress/wp-admin/themes.php
http://www.2rosenthals.net/wordpress/wp-admin/tools.php
http://www.2rosenthals.net/wordpress/wp-admin/update-core.php
http://www.2rosenthals.net/wordpress/wp-admin/update-core.php?action=do-core-upgrade
http://www.2rosenthals.net/wordpress/wp-admin/update-core.php?action=do-plugin-upgrade
http://www.2rosenthals.net/wordpress/wp-admin/update.php?action=install-plugin&plugin=wp-post-to-pdf&_wpnonce=99a20360a6
http://www.2rosenthals.net/wordpress/wp-admin/update.php?action=update-selected&plugins=akismet%2Fakismet.php&_wpnonce=e5c5f98ae2
http://www.2rosenthals.net/wordpress/wp-admin/update.php?action=update-selected&plugins=postie%2Fpostie.php&_wpnonce=9301f7883a
http://www.2rosenthals.net/wordpress/wp-admin/update.php?action=update-selected&plugins=ultimate-tinymce%2Fmain.php%2Cyet-another-related-posts-plugin%2Fyarpp.php&_wpnonce=540719f73e
http://www.2rosenthals.net/wordpress/wp-admin/update.php?action=update-selected&plugins=ultimate-tinymce%2Fmain.php&_wpnonce=5be9516d74
http://www.2rosenthals.net/wordpress/wp-admin/upgrade.php?step=1&backto=%2Fwordpress%2Fwp-admin%2Fupdate-core.php
http://www.2rosenthals.net/wordpress/wp-admin/upgrade.php?_wp_http_referer=%2Fwordpress%2Fwp-admin%2Fupdate-core.php
http://www.2rosenthals.net/wordpress/wp-content/plugins/akismet/akismet.css?ver=2.5.4.4
http://www.2rosenthals.net/wordpress/wp-content/plugins/akismet/akismet.js?ver=2.5.4.6
http://www.2rosenthals.net/wordpress/wp-content/plugins/auto-syntaxhighlighter/SyntaxHighlighter/build/scripts/shBrushBash-min.js
http://www.2rosenthals.net/wordpress/wp-content/plugins/auto-syntaxhighlighter/SyntaxHighlighter/build/scripts/shBrushPlain-min.js
http://www.2rosenthals.net/wordpress/wp-content/plugins/auto-syntaxhighlighter/SyntaxHighlighter/build/scripts/shBrushXml-min.js
http://www.2rosenthals.net/wordpress/wp-content/plugins/auto-syntaxhighlighter/SyntaxHighlighter/build/styles/shThemeDefault-min.css?ver=3.0.83
http://www.2rosenthals.net/wordpress/wp-content/plugins/auto-syntaxhighlighter/tinymce/ash/ash_editor.html?ver=2.3.3&ver=358-23224
http://www.2rosenthals.net/wordpress/wp-content/plugins/auto-syntaxhighlighter/tinymce/ash/ash_fullscreen.css?ver=2.3.3
http://www.2rosenthals.net/wordpress/wp-content/plugins/auto-syntaxhighlighter/tinymce/ash/editor_plugin.js?ver=358-23224
http://www.2rosenthals.net/wordpress/wp-content/plugins/auto-syntaxhighlighter/tinymce/ash/img/ash.gif
http://www.2rosenthals.net/wordpress/wp-content/plugins/auto-syntaxhighlighter/tinymce/ash/js/ash.js?ver=2.3
http://www.2rosenthals.net/wordpress/wp-content/plugins/better-plugin-compatibility-control/img/info.gif
http://www.2rosenthals.net/wordpress/wp-content/plugins/better-plugin-compatibility-control/js/bbpc_dom.3.1.js?ver=3.5
http://www.2rosenthals.net/wordpress/wp-content/plugins/better-plugin-compatibility-control/js/bbpc_dom.3.1.js?ver=3.5.1
http://www.2rosenthals.net/wordpress/wp-content/plugins/changelogger/css/style.css?ver=1.2.15
http://www.2rosenthals.net/wordpress/wp-content/plugins/changelogger/js/admin_scripts.js?ver=1.2.15
http://www.2rosenthals.net/wordpress/wp-content/plugins/ckeditor-for-wordpress/ckeditor/ckeditor.js?t=CBDD&ver=3.5.1
http://www.2rosenthals.net/wordpress/wp-content/plugins/ckeditor-for-wordpress/ckeditor/ckeditor.js?ver=3.4.2
http://www.2rosenthals.net/wordpress/wp-content/plugins/ckeditor-for-wordpress/ckeditor/ckeditor.js?ver=3.5.1
http://www.2rosenthals.net/wordpress/wp-content/plugins/ckeditor-for-wordpress/ckeditor.config.js?t=CBDD
http://www.2rosenthals.net/wordpress/wp-content/plugins/ckeditor-for-wordpress/ckeditor/contents.css
http://www.2rosenthals.net/wordpress/wp-content/plugins/ckeditor-for-wordpress/ckeditor/lang/en.js?t=CBDD
http://www.2rosenthals.net/wordpress/wp-content/plugins/ckeditor-for-wordpress/ckeditor/plugins/fakeobjects/images/spacer.gif?t=CBDD
http://www.2rosenthals.net/wordpress/wp-content/plugins/ckeditor-for-wordpress/ckeditor/plugins/icons.png?t=CBDD
http://www.2rosenthals.net/wordpress/wp-content/plugins/ckeditor-for-wordpress/ckeditor/plugins/smiley/images/whatchutalkingabout_smile.gif
http://www.2rosenthals.net/wordpress/wp-content/plugins/ckeditor-for-wordpress/ckeditor/plugins/smiley/images/wink_smile.gif
http://www.2rosenthals.net/wordpress/wp-content/plugins/ckeditor-for-wordpress/ckeditor/skins/kama/editor.css?t=CBDD
http://www.2rosenthals.net/wordpress/wp-content/plugins/ckeditor-for-wordpress/ckeditor/skins/kama/icons.png
http://www.2rosenthals.net/wordpress/wp-content/plugins/ckeditor-for-wordpress/ckeditor/skins/kama/skin.js?t=CBDD
http://www.2rosenthals.net/wordpress/wp-content/plugins/ckeditor-for-wordpress/ckeditor.styles.js?t=CBDD
http://www.2rosenthals.net/wordpress/wp-content/plugins/ckeditor-for-wordpress/images/ckeditor_ico16.png
http://www.2rosenthals.net/wordpress/wp-content/plugins/ckeditor-for-wordpress/images/ckeditor_ico32.png
http://www.2rosenthals.net/wordpress/wp-content/plugins/ckeditor-for-wordpress/images/compat_logos.png
http://www.2rosenthals.net/wordpress/wp-content/plugins/ckeditor-for-wordpress/images/configuration.png
http://www.2rosenthals.net/wordpress/wp-content/plugins/ckeditor-for-wordpress/images/documentation.png
http://www.2rosenthals.net/wordpress/wp-content/plugins/ckeditor-for-wordpress/images/exclamation.png
http://www.2rosenthals.net/wordpress/wp-content/plugins/ckeditor-for-wordpress/images/help.png
http://www.2rosenthals.net/wordpress/wp-content/plugins/ckeditor-for-wordpress/images/menuicon.gif
http://www.2rosenthals.net/wordpress/wp-content/plugins/ckeditor-for-wordpress/images/plugin.png
http://www.2rosenthals.net/wordpress/wp-content/plugins/ckeditor-for-wordpress/includes/basic.js?ver=3.5.1
http://www.2rosenthals.net/wordpress/wp-content/plugins/ckeditor-for-wordpress/includes/ckeditor.comment-reply.js?ver=20100901
http://www.2rosenthals.net/wordpress/wp-content/plugins/ckeditor-for-wordpress/includes/ckeditor.utils.js?ver=3.4.2
http://www.2rosenthals.net/wordpress/wp-content/plugins/ckeditor-for-wordpress/includes/ckeditor.utils.js?ver=3.5
http://www.2rosenthals.net/wordpress/wp-content/plugins/ckeditor-for-wordpress/includes/ckeditor.utils.js?ver=3.5.1
http://www.2rosenthals.net/wordpress/wp-content/plugins/ckeditor-for-wordpress/includes/overview.css?ver=4.0
http://www.2rosenthals.net/wordpress/wp-content/plugins/ckeditor-for-wordpress/plugins/wpgallery/plugin.js?t=CBDD
http://www.2rosenthals.net/wordpress/wp-content/plugins/ckeditor-for-wordpress/plugins/wpmore/images/more_bug.gif?t=CBDD
http://www.2rosenthals.net/wordpress/wp-content/plugins/ckeditor-for-wordpress/plugins/wpmore/images/more.gif?t=CBDD
http://www.2rosenthals.net/wordpress/wp-content/plugins/ckeditor-for-wordpress/plugins/wpmore/plugin.js?t=CBDD
http://www.2rosenthals.net/wordpress/wp-content/plugins/contact-form-7/admin/images/menu-icon.png
http://www.2rosenthals.net/wordpress/wp-content/plugins/contact-form-7/includes/css/styles.css?ver=3.3.3
http://www.2rosenthals.net/wordpress/wp-content/plugins/contact-form-7/includes/js/jquery.form.min.js?ver=3.25.0-2013.01.18
http://www.2rosenthals.net/wordpress/wp-content/plugins/contact-form-7/includes/js/scripts.js?ver=3.3.1
http://www.2rosenthals.net/wordpress/wp-content/plugins/contact-form-7/includes/js/scripts.js?ver=3.3.2
http://www.2rosenthals.net/wordpress/wp-content/plugins/contact-form-7/includes/js/scripts.js?ver=3.3.3
http://www.2rosenthals.net/wordpress/wp-content/plugins/link-library/icons/folder-beige-internet-icon.png
http://www.2rosenthals.net/wordpress/wp-content/plugins/mtc-ckeditor-link-page/ckeplugin/plugin.js?t=CBDD
http://www.2rosenthals.net/wordpress/wp-content/plugins/post-type-switcher/js/quickedit.js?ver=3.5.1
http://www.2rosenthals.net/wordpress/wp-content/plugins/subscribe2/include/email_edit.png
http://www.2rosenthals.net/wordpress/wp-content/plugins/subscribe2/include/s2_button.png
http://www.2rosenthals.net/wordpress/wp-content/plugins/subscribe2/tinymce3/css/content.css?ver=358-23224
http://www.2rosenthals.net/wordpress/wp-content/plugins/subscribe2/tinymce3/editor_plugin.min.js?ver=358-23224
http://www.2rosenthals.net/wordpress/wp-content/plugins/syntaxhighlighter-ckeditor-button/syntaxhighlight/dialogs/syntaxhighlight.js?t=CBDD
http://www.2rosenthals.net/wordpress/wp-content/plugins/syntaxhighlighter-ckeditor-button/syntaxhighlight/images/syntaxhighlight.gif?t=CBDD
http://www.2rosenthals.net/wordpress/wp-content/plugins/syntaxhighlighter-ckeditor-button/syntaxhighlight/lang/en.js?t=CBDD
http://www.2rosenthals.net/wordpress/wp-content/plugins/syntaxhighlighter-ckeditor-button/syntaxhighlight/plugin.js?t=CBDD
http://www.2rosenthals.net/wordpress/wp-content/plugins/syntax-highlighter-compress/scripts/shAutoloader.js
http://www.2rosenthals.net/wordpress/wp-content/plugins/syntax-highlighter-compress/scripts/shBrushCss.js
http://www.2rosenthals.net/wordpress/wp-content/plugins/syntax-highlighter-compress/scripts/shBrushPhp.js
http://www.2rosenthals.net/wordpress/wp-content/plugins/syntax-highlighter-compress/scripts/shBrushXml.js
http://www.2rosenthals.net/wordpress/wp-content/plugins/syntax-highlighter-compress/scripts/shCore.js
http://www.2rosenthals.net/wordpress/wp-content/plugins/syntax-highlighter-compress/styles/shCoreDefault.css
http://www.2rosenthals.net/wordpress/wp-content/plugins/syntax-highlighter-compress/tinymce/editor_plugin.js?ver=358-23224
http://www.2rosenthals.net/wordpress/wp-content/plugins/syntax-highlighter-compress/tinymce/shc.gif
http://www.2rosenthals.net/wordpress/wp-content/plugins/syntax-highlighter-compress/tinymce/window.php?ver=358-23224
http://www.2rosenthals.net/wordpress/wp-content/plugins/theme-my-login/modules/themed-profiles/themed-profiles.css?ver=3.5.1
http://www.2rosenthals.net/wordpress/wp-content/plugins/theme-my-login/theme-my-login.css?ver=6.2.3
http://www.2rosenthals.net/wordpress/wp-content/plugins/theme-my-login/theme-my-login.css?ver=6.3.8
http://www.2rosenthals.net/wordpress/wp-content/plugins/ultimate-tinymce/emotions/img/popobig/beat_ball.png
http://www.2rosenthals.net/wordpress/wp-content/plugins/wordpress-admin-bar-improved/wpabi.css?ver=2.0
http://www.2rosenthals.net/wordpress/wp-content/plugins/wordpress-admin-bar-improved/wpabi.js?ver=1.0
http://www.2rosenthals.net/wordpress/wp-content/plugins/wp-appstore/assets/jquery.tools.min.js?ver=20110322
http://www.2rosenthals.net/wordpress/wp-content/plugins/wp-appstore/assets/scrollable-buttons.css?ver=20110322
http://www.2rosenthals.net/wordpress/wp-content/plugins/wp-appstore/assets/scrollable-horizontal.css?ver=20110322
http://www.2rosenthals.net/wordpress/wp-content/plugins/wp-appstore/wp-appstore.css?ver=20110322
http://www.2rosenthals.net/wordpress/wp-content/plugins/wp-maintenance-mode/css/overcast/images/ui-bg_inset-soft_50_c9c9c9_1x100.png
http://www.2rosenthals.net/wordpress/wp-content/plugins/wp-maintenance-mode/css/overcast/jquery-ui-1.8.21.custom.css?ver=3.5
http://www.2rosenthals.net/wordpress/wp-content/plugins/wp-maintenance-mode/css/overcast/jquery-ui-1.8.21.custom.css?ver=3.5.1
http://www.2rosenthals.net/wordpress/wp-content/plugins/wp-maintenance-mode/css/style.css?ver=3.4.2
http://www.2rosenthals.net/wordpress/wp-content/plugins/wp-maintenance-mode/css/style.css?ver=3.5.1
http://www.2rosenthals.net/wordpress/wp-content/plugins/wp-maintenance-mode/js/jquery-ui-timepicker-addon.js?ver=02-22-2013
http://www.2rosenthals.net/wordpress/wp-content/plugins/wp-maintenance-mode/js/wp-maintenance-mode.js?ver=3.5
http://www.2rosenthals.net/wordpress/wp-content/plugins/wp-maintenance-mode/js/wp-maintenance-mode.js?ver=3.5.1
http://www.2rosenthals.net/wordpress/wp-content/plugins/wp-post-to-pdf/asset/css/admin.css?ver=1.0
http://www.2rosenthals.net/wordpress/wp-content/plugins/wp-post-to-pdf/asset/images/pdf.png
http://www.2rosenthals.net/wordpress/wp-content/plugins/wp-post-to-pdf/asset/images/toggle.gif
http://www.2rosenthals.net/wordpress/wp-content/plugins/wptouch/images/good.png
http://www.2rosenthals.net/wordpress/wp-content/plugins/wptouch/images/icon-pool/Contacts.png
http://www.2rosenthals.net/wordpress/wp-content/plugins/wptouch/images/icon-pool/Home.png
http://www.2rosenthals.net/wordpress/wp-content/plugins/wptouch/images/icon-pool/Mail.png
http://www.2rosenthals.net/wordpress/wp-content/plugins/wptouch/images/icon-pool/Notes.png
http://www.2rosenthals.net/wordpress/wp-content/plugins/wptouch/images/icon-pool/RSS.png
http://www.2rosenthals.net/wordpress/wp-content/plugins/wptouch/images/icon-pool/ToDo.png
http://www.2rosenthals.net/wordpress/wp-content/plugins/wptouch/themes/core/core-css/wptouch-switch-link.css
http://www.2rosenthals.net/wordpress/wp-content/plugins/wptouch/themes/core/core-images/backgrounds/low_contrast_linen.png
http://www.2rosenthals.net/wordpress/wp-content/plugins/wptouch/themes/core/core-images/bookmarks/digg.jpg
http://www.2rosenthals.net/wordpress/wp-content/plugins/wptouch/themes/core/core-images/bookmarks/magnolia.jpg
http://www.2rosenthals.net/wordpress/wp-content/plugins/wptouch/themes/core/core-images/bookmarks/reddit.jpg
http://www.2rosenthals.net/wordpress/wp-content/plugins/wptouch/themes/core/core-images/bookmarks/technorati.jpg
http://www.2rosenthals.net/wordpress/wp-content/plugins/wptouch/themes/core/core-images/head-close.png
http://www.2rosenthals.net/wordpress/wp-content/plugins/wptouch/themes/core/core-images/head-fade-bk.png
http://www.2rosenthals.net/wordpress/wp-content/plugins/wptouch/themes/core/core-images/menu-sprite.png
http://www.2rosenthals.net/wordpress/wp-content/plugins/wptouch/themes/default/style.css
http://www.2rosenthals.net/wordpress/wp-content/plugins/yet-another-related-posts-plugin/js/metabox.js?ver=4.0.6
http://www.2rosenthals.net/wordpress/wp-content/plugins/youtube-embed/css/admin.min.css?ver=3.5.1
http://www.2rosenthals.net/wordpress/wp-content/plugins/youtube-embed/css/aye-admin.min.css?ver=3.5
http://www.2rosenthals.net/wordpress/wp-content/plugins/youtube-embed/css/aye-main.min.css?ver=3.4.2
http://www.2rosenthals.net/wordpress/wp-content/plugins/youtube-embed/css/aye-main.min.css?ver=3.5
http://www.2rosenthals.net/wordpress/wp-content/plugins/youtube-embed/css/main.min.css?ver=3.5.1
http://www.2rosenthals.net/wordpress/wp-content/plugins/youtube-embed/images/admin_menu_icon.png
http://www.2rosenthals.net/wordpress/wp-content/plugins/youtube-embed/images/menu_icon.png
http://www.2rosenthals.net/wordpress/wp-content/plugins/youtube-embed/images/youtube_button_b&w.png
http://www.2rosenthals.net/wordpress/wp-content/plugins/youtube-embed/js/mce-button.min.php?ver=358-23224
http://www.2rosenthals.net/wordpress/wp-content/themes/carrington-mobile-1.0.2/screenshot.png
http://www.2rosenthals.net/wordpress/wp-content/themes/classic/screenshot.png
http://www.2rosenthals.net/wordpress/wp-content/themes/default/screenshot.png
http://www.2rosenthals.net/wordpress/wp-content/themes/leathernote/screenshot.png
http://www.2rosenthals.net/wordpress/wp-content/themes/lightword-2.0.0.6/screenshot.png
http://www.2rosenthals.net/wordpress/wp-content/themes/lightword-child/screenshot.png
http://www.2rosenthals.net/wordpress/wp-content/themes/lightword-child/style.css?ver=1.0.0.0
http://www.2rosenthals.net/wordpress/wp-content/themes/lightword/images/date_comm_box.png
http://www.2rosenthals.net/wordpress/wp-content/themes/lightword/images/nav.png
http://www.2rosenthals.net/wordpress/wp-content/themes/lightword/images/older_newer.png
http://www.2rosenthals.net/wordpress/wp-content/themes/lightword/images/rss.png
http://www.2rosenthals.net/wordpress/wp-content/themes/lightword/images/searchbox.png
http://www.2rosenthals.net/wordpress/wp-content/themes/lightword/images/sidebar_h3.png
http://www.2rosenthals.net/wordpress/wp-content/themes/lightword/images/submit_btn.png
http://www.2rosenthals.net/wordpress/wp-content/themes/lightword/images/wider/content_bottom.gif
http://www.2rosenthals.net/wordpress/wp-content/themes/lightword/images/wider/content_bottom.png
http://www.2rosenthals.net/wordpress/wp-content/themes/lightword/images/wider/content_middle.png
http://www.2rosenthals.net/wordpress/wp-content/themes/lightword/images/wider/content_top.png
http://www.2rosenthals.net/wordpress/wp-content/themes/lightword/images/wider/tags_category.png
http://www.2rosenthals.net/wordpress/wp-content/themes/lightword/js/cufon-yui.js
http://www.2rosenthals.net/wordpress/wp-content/themes/lightword/js/menu.js
http://www.2rosenthals.net/wordpress/wp-content/themes/lightword/js/tabs.js
http://www.2rosenthals.net/wordpress/wp-content/themes/lightword/js/vera.font.js
http://www.2rosenthals.net/wordpress/wp-content/themes/lightword/style.css
http://www.2rosenthals.net/wordpress/wp-content/themes/lightword/wider.css?ver=1.0.0.0
http://www.2rosenthals.net/wordpress/wp-content/themes/twentyeleven/screenshot.png
http://www.2rosenthals.net/wordpress/wp-content/uploads/2011/01/net_wan_cloud.svg_.hi_-300x158.png
http://www.2rosenthals.net/wordpress/wp-content/uploads/2011/01/net_wan_cloud.svg_.hi_.png
http://www.2rosenthals.net/wordpress/wp-content/uploads/2011/11/LewisR_avatar-16x16.jpg
http://www.2rosenthals.net/wordpress/wp-content/uploads/2011/11/LewisR_avatar-50x50.jpg
http://www.2rosenthals.net/wordpress/wp-content/uploads/2011/11/LewisR_avatar-64x64.jpg
http://www.2rosenthals.net/wordpress/wp-content/uploads/2011/11/LewisR_avatar.jpg
http://www.2rosenthals.net/wordpress/wp-content/uploads/2012/05/PDXESES_Embassy_Suites_Portland-Airport_gallery_welcome-300x234.jpg
http://www.2rosenthals.net/wordpress/wp-content/uploads/2012/07/WPTouch_ShareBar_modified.png
http://www.2rosenthals.net/wordpress/wp-content/uploads/2012/10/BetterPrivacy-logo.png
http://www.2rosenthals.net/wordpress/wp-content/uploads/2012/11/Writer-Columns-properties.jpg
http://www.2rosenthals.net/wordpress/wp-content/uploads/wpcf7_captcha/1147162443.png
http://www.2rosenthals.net/wordpress/wp-content/uploads/wpcf7_captcha/1343786788.png
http://www.2rosenthals.net/wordpress/wp-content/uploads/wpcf7_captcha/154246084.png
http://www.2rosenthals.net/wordpress/wp-content/uploads/wpcf7_captcha/1695641856.png
http://www.2rosenthals.net/wordpress/wp-content/uploads/wpcf7_captcha/514117729.png
http://www.2rosenthals.net/wordpress/wp-content/uploads/wpcf7_captcha/542904053.png
http://www.2rosenthals.net/wordpress/wp-content/uploads/wpcf7_captcha/826095337.png
http://www.2rosenthals.net/wordpress/wp-content/uploads/wpcf7_captcha/941254618.png
http://www.2rosenthals.net/wordpress/wp-includes/css/admin-bar.min.css?ver=3.5
http://www.2rosenthals.net/wordpress/wp-includes/css/admin-bar.min.css?ver=3.5.1
http://www.2rosenthals.net/wordpress/wp-includes/css/buttons.min.css?ver=3.5
http://www.2rosenthals.net/wordpress/wp-includes/css/editor.css?ver=3.4.2
http://www.2rosenthals.net/wordpress/wp-includes/css/editor.min.css?ver=3.5
http://www.2rosenthals.net/wordpress/wp-includes/css/editor.min.css?ver=3.5.1
http://www.2rosenthals.net/wordpress/wp-includes/images/admin-bar-sprite-2x.png?d=20120830
http://www.2rosenthals.net/wordpress/wp-includes/images/admin-bar-sprite.png?d=20120830
http://www.2rosenthals.net/wordpress/wp-includes/images/blank.gif
http://www.2rosenthals.net/wordpress/wp-includes/images/down_arrow.gif
http://www.2rosenthals.net/wordpress/wp-includes/images/smilies/icon_wink.gif
http://www.2rosenthals.net/wordpress/wp-includes/images/wpicons.png?ver=20120720
http://www.2rosenthals.net/wordpress/wp-includes/js/admin-bar.min.js?ver=3.5.1
http://www.2rosenthals.net/wordpress/wp-includes/js/comment-reply.min.js?ver=3.5.1
http://www.2rosenthals.net/wordpress/wp-includes/js/jquery/jquery.js?ver=1.7.2
http://www.2rosenthals.net/wordpress/wp-includes/js/jquery/jquery.js?ver=1.8.3
http://www.2rosenthals.net/wordpress/wp-includes/js/jquery/ui/jquery.ui.button.min.js?ver=1.9.2
http://www.2rosenthals.net/wordpress/wp-includes/js/jquery/ui/jquery.ui.core.min.js?ver=1.9.2
http://www.2rosenthals.net/wordpress/wp-includes/js/jquery/ui/jquery.ui.dialog.min.js?ver=1.8.20
http://www.2rosenthals.net/wordpress/wp-includes/js/jquery/ui/jquery.ui.dialog.min.js?ver=1.9.2
http://www.2rosenthals.net/wordpress/wp-includes/js/jquery/ui/jquery.ui.draggable.min.js?ver=1.9.2
http://www.2rosenthals.net/wordpress/wp-includes/js/jquery/ui/jquery.ui.mouse.min.js?ver=1.9.2
http://www.2rosenthals.net/wordpress/wp-includes/js/jquery/ui/jquery.ui.position.min.js?ver=1.8.20
http://www.2rosenthals.net/wordpress/wp-includes/js/jquery/ui/jquery.ui.position.min.js?ver=1.9.2
http://www.2rosenthals.net/wordpress/wp-includes/js/jquery/ui/jquery.ui.resizable.min.js?ver=1.9.2
http://www.2rosenthals.net/wordpress/wp-includes/js/jquery/ui/jquery.ui.widget.min.js?ver=1.9.2
http://www.2rosenthals.net/wordpress/wp-includes/js/thickbox/loadingAnimation.gif
http://www.2rosenthals.net/wordpress/wp-includes/js/thickbox/thickbox.css?ver=20121105
http://www.2rosenthals.net/wordpress/wp-includes/js/thickbox/thickbox.css?ver=3.4.2
http://www.2rosenthals.net/wordpress/wp-includes/js/tinymce/langs/wp-langs-en.js?ver=358-23224
http://www.2rosenthals.net/wordpress/wp-includes/js/tinymce/plugins/inlinepopups/skins/clearlooks2/img/buttons.gif
http://www.2rosenthals.net/wordpress/wp-includes/js/tinymce/plugins/inlinepopups/skins/clearlooks2/img/corners.gif
http://www.2rosenthals.net/wordpress/wp-includes/js/tinymce/plugins/inlinepopups/skins/clearlooks2/img/drag.gif
http://www.2rosenthals.net/wordpress/wp-includes/js/tinymce/plugins/inlinepopups/skins/clearlooks2/img/horizontal.gif
http://www.2rosenthals.net/wordpress/wp-includes/js/tinymce/plugins/inlinepopups/skins/clearlooks2/window.css?ver=358-23224
http://www.2rosenthals.net/wordpress/wp-includes/js/tinymce/plugins/spellchecker/css/content.css
http://www.2rosenthals.net/wordpress/wp-includes/js/tinymce/plugins/wordpress/img/trans.gif
http://www.2rosenthals.net/wordpress/wp-includes/js/tinymce/plugins/wpeditimage/img/delete.png
http://www.2rosenthals.net/wordpress/wp-includes/js/tinymce/plugins/wpeditimage/img/image.png
http://www.2rosenthals.net/wordpress/wp-includes/js/tinymce/plugins/wpgallery/img/delete.png
http://www.2rosenthals.net/wordpress/wp-includes/js/tinymce/plugins/wpgallery/img/edit.png
http://www.2rosenthals.net/wordpress/wp-includes/js/tinymce/themes/advanced/img/icons.gif
http://www.2rosenthals.net/wordpress/wp-includes/js/tinymce/themes/advanced/skins/wp_theme/content.css
http://www.2rosenthals.net/wordpress/wp-includes/js/tinymce/themes/advanced/skins/wp_theme/dialog.css?ver=358-23224
http://www.2rosenthals.net/wordpress/wp-includes/js/tinymce/themes/advanced/skins/wp_theme/img/more_bug.gif
http://www.2rosenthals.net/wordpress/wp-includes/js/tinymce/themes/advanced/skins/wp_theme/img/tabs.gif
http://www.2rosenthals.net/wordpress/wp-includes/js/tinymce/tiny_mce_popup.js
http://www.2rosenthals.net/wordpress/wp-includes/js/tinymce/tiny_mce_popup.js?ver=345-20110908
http://www.2rosenthals.net/wordpress/wp-includes/js/tinymce/utils/form_utils.js
http://www.2rosenthals.net/wordpress/wp-includes/js/tinymce/utils/form_utils.js?ver=345-20110908
http://www.2rosenthals.net/wordpress/wp-includes/js/tinymce/utils/mctabs.js
http://www.2rosenthals.net/wordpress/wp-includes/js/tinymce/wp-tinymce.php?c=1&ver=358-23224
http://www.2rosenthals.net/wordpress/wp-includes/js/tw-sack.js?ver=1.6.1
http://www.2rosenthals.net/wordpress/wp-includes/js/tw-sack.min.js?ver=1.6.1
http://www.2rosenthals.net/wordpress/wp-login.php
http://www.2rosenthals.net/wordpress/?wptouch_view=normal&wptouch_redirect_nonce=0d414bafc5&wptouch_redirect=%2Fwordpress%2Fbetterprivacy-mod-for-seamonkey-532%2F
http://www.2rosenthals.net/wordpress/wrestling-with-apaches-mod_rewrite-374/
http://www.2rosenthals.net/wordpress/xmlrpc.php
http://www.2rosenthals.net/wordpress/xmlrpc.php?rsd
siegeurls1 (78,190 bytes)

psmedley

2015-01-02 15:49

administrator   ~0002993

Is it worth also posting the relevant php.ini / httpd.conf here?

LewisR

2015-01-02 15:58

developer  

php.ini (69,783 bytes)

LewisR

2015-01-02 16:03

developer  

httpd.conf (20,298 bytes)

LewisR

2015-01-02 16:03

developer  

httpd-mpm.conf (4,606 bytes)

LewisR

2015-01-02 16:04

developer  

httpd-php.conf (862 bytes)

LewisR

2015-01-02 16:07

developer   ~0002994

Uploaded php.ini, httpd.conf, httpd-mpm.conf, and httpd-php.conf.

httpd-ssl.conf and httpd-vhosts.conf contain a bunch of stuff which I would prefer to not have to edit for public consumption, and other icnluded confs (see bottom of httpd.conf) are irrelevant for these purposes.

Also note that php.ini loads the pdf module, but testing shows that this is not relevant to success or failure when testing against php pages. ;-)

psmedley

2019-09-29 10:41

administrator   ~0003339

is this ticket still adding value? or should it be closed and new ones raised targeting the remaining issues?

Issue History

Date Modified Username Field Change
2014-12-20 08:59 LewisR New Issue
2014-12-20 09:01 LewisR Note Added: 0002940
2014-12-20 09:09 LewisR Note Added: 0002941
2014-12-20 09:15 LewisR Note Added: 0002942
2014-12-20 09:15 LewisR File Added: testing-http-php-with-os2trace.txt
2014-12-20 09:19 LewisR File Added: testing-http-php-with-os2trace-20141108.txt
2014-12-20 09:20 LewisR Note Edited: 0002942 View Revisions
2014-12-20 09:22 LewisR Note Added: 0002943
2014-12-20 09:36 LewisR Note Added: 0002944
2014-12-20 09:36 LewisR File Added: readwrite.c.diff
2014-12-20 09:38 LewisR File Added: readwrite2.c.diff
2014-12-20 09:38 LewisR Note Edited: 0002944 View Revisions
2014-12-20 09:42 LewisR Note Added: 0002945
2014-12-20 09:46 LewisR Note Added: 0002946
2014-12-20 09:46 LewisR File Added: readwrite3.c.diff
2014-12-20 09:53 LewisR Note Added: 0002947
2014-12-20 09:58 LewisR Note Added: 0002948
2014-12-20 10:06 LewisR Note Added: 0002949
2014-12-20 10:12 LewisR Note Added: 0002950
2014-12-20 10:12 LewisR File Added: readwrite4.c.diff
2014-12-20 10:19 LewisR Note Added: 0002951
2014-12-20 10:20 LewisR File Added: open.c.diff
2014-12-20 10:20 LewisR File Added: pipe.c.diff
2014-12-20 10:26 LewisR Note Added: 0002952
2014-12-20 10:47 psmedley File Deleted: testing-http-php-with-os2trace.txt
2014-12-23 09:51 Steven Levine Relationship added related to 0000638
2015-01-02 07:32 psmedley Note Added: 0002985
2015-01-02 07:48 LewisR Note Added: 0002986
2015-01-02 07:48 LewisR File Added: siegeurls1
2015-01-02 15:49 psmedley Note Added: 0002993
2015-01-02 15:58 LewisR File Added: php.ini
2015-01-02 16:03 LewisR File Added: httpd.conf
2015-01-02 16:03 LewisR File Added: httpd-mpm.conf
2015-01-02 16:04 LewisR File Added: httpd-php.conf
2015-01-02 16:07 LewisR Note Added: 0002994
2019-09-29 10:41 psmedley Note Added: 0003339