How to use Dave Blaschke's os2trace to test apache httpd and php (cgi) 2014-11-07 SHL Baseline 2014-11-07 SHL Rework to use alternate ServerRoot directory. 2014-11-08 SHL Document logging level 2014-11-08 SHL Preset DOSCALLS APIs selections. == Prerequisites == - 4OS2 - Dave Blaschke's os2trace package. Available from http://www.angelfire.com/tx4/blaschke/OS2Trace/ - My os2traceenv.cmd script. Available from www.warpcave.com/betas - A working apache2 setup. - A working php CGI setup. == Files that will be modified == Os2trace modifies executables to insert forwarders to the tracing routines. The following files will need to be modified httpd.dll php.exe (cgi version) php5.dll libc066.dll == Install os2traceenv.cmd script == - Install os2traceenv.cmd to a directory in the PATH. == Install os2trace == - Create a directory for os2trace and unzip the distribution. I use d:\devtools\os2trace, but you can use a directory of your choice. == Setup a mirror of your production apache2/php installation == - Mirror your apache2 ServerRoot tree to a work directory. This makes setup simplifier and leaves the production apache's setup unsullied. I used d:\Internet\apache22-trc as the root, but you can use a directory of your choice. - Copy php5.dll to the mirror's bin directory. This makes BEGINLIBPATH setup simpler. - If your production cgi-bin is not apache2\cgi-bin, create a cgi-bin directory in the mirror and copy content of the production cgi-bin to the mirror's cgi-bin. - Edit the mirror's httpd.conf to match the new directory locations. Update ServerRoot to point to the mirror. Update ScriptAlias to point to the mirror's cgi-bin. Update absolute paths that should not point at production directories. Update log paths to leave the production logs unsullied. Update Directory sections as needed. Etc. - Copy libc066 DLLs to bin directory. This makes BEGINLIBPATH setup simpler. - Open a 4OS2 VIO session - Change to your os2trace directory and run - Start apache from the bin directory with httpd -d.. to test your edits. - Apache should start a serve content. - Stop apache. - Do not close the VIO session. You will continue to need it. == Modify files for os2trace == - Change to your os2trace directory and run os2traceenv to set up the os2trace environment. os2traceenv updates PATH and BEGINLIBPATH. - Change back to your mirror's bin directory - Run pmos2trc - Set the logging level to level 3 From the Logging Level dialog, select the Log API parameter contents (level 3) information item. Click OK to confirm the settings. The settings are stored in os2.ini in the os2trace application and will persist until changed. - Customize the DOSCALLS APIs selections. Use the Customize -> DOSCALLS APIs menu option to tune the settings. From the DOSCALLS APIs dialog, turn on FILE PIPE and PROC and turn off all the other selections. Click OK to confirm the settings. The settings are stored in os2.ini in the os2trace application and will persist until changed. - Enable tracing for httpd.dll php.exe (cgi version) php5.dll libc066.dll For each file, use the Enable -> Open file menu option to select the file to modify. From the Enablement dialog turn on DOSCALLS and disable all others Click OK to confirm the settings. - After all the files have been enabled for tracing, close pmos2trc leaving the VIO session open. == Adjust PATHs == - Edit BEGINLIBPATH and the mirror's bin directory so that apache2 and php can find the modified php5.dll and the modified libc066.dll when running LIBPATHSTRICT. - On my setup this results in BEGINLIBPATH=d:\internet\apache22-trc\bin;d:\devtools\os2trace; - Run set LIBPATHSTRICT=T to ensure that the modified libc066.dll is used. == Start apache and verify os2trace operation == - Start apache from the mirror's bin directory with httpd -d.. - If all goes well, apache2 will start and os2trace will report that it generated unique .trc file names for the 2nd and 3rd httpd.exe instances. - Try a simple php test. - Verify that os2trace creates a php.trc file in the mirror's cgi-bin directory. - Review the content of the generated .trc files. You should be able to see the pipe IO and file IO operations. You should also be able to see the CGI data that was read and written. == Testing with siege == - When running siege, os2trace will generate unique .trc file names for secondary php.exe instances. The primary php.trc file will contain reports for the php.exe instances that happened to be started when no other php.exe instances were active. - The size of the .trc files will depend on the amount of IO done by the PHP code. - It is recommended that you turn off as much logging as possible. This will reduce the amount of IO logged to the .trc files. The siege test should not need any CustomLogs and LogLevel Error should be sufficient for our purposes. - The httpd.trc file can get large, as can the php.trc file because they will contain the output for multiple siege transactions. - A siege run will create a large number of php .trc files unless the siege dies quickly. Each GET request for PHP content will spawn a new PHP instance. - It's probably a good idea to stop httpd between siege runs and clean up any left over .trc files. If a siege run fails, the .trc files should be moved to uniquely name archive directory for post-mortem analysis. Otherwise, the .trc files can be deleted. == Post mortem analysis == - Once a siege run produces some hung php.exe instances, we are ready to analyze. - Find the php .trc files that are missing the final "stopping at" message. These correspond to the hung instances. Verify that the PIDs match what pstat /c | grep PHP reports. - For each hung php instance, examine the httpd .trc files and find the Dos32ExecPgm invocation that started this php instances. - Note the PID/TID combo and search for API failures or unexpected operations on this thread. - more later ... ============================================================ ==== Old setup method using saved copies of executables === ============================================================ == Save copies to files that will be modified by os2trace == - You will be instrumenting httpd.dll php.exe (cgi version) php5.dll libc066.dll os2trace can remove the modification, but I prefer to restore saved copies == Modify files for os2trace == - Open a 4OS2 VIO session - Change to your os2trace directory and run os2traceenv to set up an os2trace environment. os2traceenv updates PATH and BEGINLIBPATH. - Run pmos2trc and enable tracing for httpd.dll php.exe (cgi version) php5.dll libc066.dll Use Enable -> Open file to select the file to modify In the Enablement dialog turn on DOSCALLS and disable all others Click OK to confirm the settings. Use Customize -> DOSCALLS APIs to tune the settings In the DOSCALLS APIs dialog, turn on FILE PIPE PROC and turn off all the other selections. Click OK to confirm the settings. - Close pmos2trc and leave the VIO session open. == Prepare to start httpd == - Change to your apache2 bin directory. - Edit BEGINLIBPATH so that apache2 and php can find the modified libc DLLs. - Edit BEGINLIBPATH (if needed) so that php.exe can find php5.dll. - On my setup this results in BEGINLIBPATH=d:\internet\0;D:\Internet\php5;d:\devtools\os2trace; - Run set LIBPATHSTRICT=T to ensure that the modified libc DLLs are used. -- This email was Anti Virus checked by Astaro Security Gateway. http://www.astaro.com