comm with (dynamic) server aborted / (first read) idle timeout

My application has a Java http client posting a single param to a
mod_fcgi ruby program; I'm hoping someone who's seen it before can
offer an instant fix ;) but I'll be happy with whatever probing
questions might help shed some light on it.

I'm using Redhat FC3 on a dual-CPU machine with Raid-5 disks and a gig
of RAM; this machine should rip, but it falls over at a post-rate of
only about 8 posts per second.  Ouch.

The FCGI application is written in ruby; I am using the latest releases
of everything, having tried all combinations of ...

   mod_fastcgi-2.4.2-1.i386.rpm 
   mod_fastcgi-2.4.2.tar.gz 
   ruby-fcgi-0.8.6.tar.gz and the new 0.8.6.1 gem file 

and using Ruby 1.8.2 compiled from sources. 

this is on the Apache/2.0.46 shipped standard with RH, and the server
also includes mod_php

    Architecture:   32-bit
    Server compiled with....
     -D APACHE_MPM_DIR="server/mpm/prefork"
     -D APR_HAS_SENDFILE
     -D APR_HAS_MMAP
     -D APR_HAVE_IPV6 (IPv4-mapped addresses disabled)
     -D APR_USE_SYSVSEM_SERIALIZE
     -D APR_USE_PTHREAD_SERIALIZE
     -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
     -D APR_HAS_OTHER_CHILD
     -D AP_HAVE_RELIABLE_PIPED_LOGS
     ...

I am seeing FastCGI apps appear to hang at the end of the processing
loop, and stay there until we hit the idle timeout.  As a result,
Apache fills the process table with dead fastcgi instances; I have
set the timeout to 10 sec so my server doesn't need restarting every
few minutes, but I get the same behaviour regardless the length of the
timeout.

the error_log reports:

 FastCGI: comm with (dynamic) server "/var/www/html/index.fcgi" aborted: 
   (first read) idle timeout (10 sec) 

 FastCGI: incomplete headers (0 bytes) received from server 
   "/var/www/html/index.fcgi" 

The fcgi application script itself is fine.  And while it is Ruby,
it's trivial:

  require 'fcgi'
  require 'timeout'

  FCGI.each_cgi {|cgi|

    begin

      timeout(5) do

        $status = "OK"
        $ack = cgi['filename']

        cgi.out('status' => $status ) {"<p>#{$ack}</p>\n\n"}

      end  #timeout

    rescue
        $stderr.print "ERROR: #{$!}"
    end
  }

Note the 5 seconds timeout wrapper -- I have tried introducing delays
to force it to expire, and it /will/ abort the program if the process
loop is tardy, but as seen above it never trips, yet under load the
application hangs for 10 seconds.

Run from the command line, this returns immediately and correctly ...

    # ruby index.fcgi
    (offline mode: enter name=value pairs on standard input)
    filename=other/other/event-summary/xt.st-event.xml
    Status: 200 OK
    Content-Type: text/html
    Content-Length: 90

    <p>other/other/event-summary/xt.st-event.xml</p>

Under normal operation, the script behaves as expected.  This script
/only/ fails when server load is moderately high, at loads of several
requests per second, less than ten; otherwise the FCGI appears to work
perfectly and resumes operation as expected when the load subsides.

I suppose it /could/ be that the listen-queue-length is sufficient
to hold docs long enough to get by whatever the natural process length
happens to be, but if that was the case I would expect to see the
CGI abort warnings in the logs even during quiet times, and I don't.
they only get 'stuck' when load is high.

The application feeding this CGI is a Java program that emits the
filenames of recently added documents, so it is also very simple;
I originally collected all the output from the webservice, but now
only collect the first line of input, then close the connection, but
it has made no difference ...

    try {

        URLConnection ws = action.openConnection();
        ws.setDoOutput(true);

        PrintWriter out = new PrintWriter(ws.getOutputStream());
        out.print( "filename=" + URLEncoder.encode(filename) );
        out.close();

        try {

            BufferedReader in = 
            new BufferedReader(
               new InputStreamReader(
                     ws.getInputStream()));

            String inline;
            if ((inline = in.readLine()) != null) {
                    cat.debug("Post ACK " + inline );
                //  response.append(inline);
            }
            in.close();

        } catch (IOException e) {
          // note the error in log4java 
        }

I've played with the following apache2 conf.d details:

    <IfModule mod_fastcgi.c>
      FastCgiIpcDir /var/run/fastcgi
      FastCgiConfig -maxClassProcesses 35 -maxProcesses 95 
                    -idle-timeout 10 -listen-queue-depth 300
      AddHandler fastcgi-script .fcgi
    </IfModule>

I've made the queue short or long, max classes high or low, no
change in the behaviour.

in Apache2, I have

    Timeout 90
    KeepAlive Off
    MaxKeepAliveRequests 100
    KeepAliveTimeout 15

    <IfModule prefork.c>
    StartServers       20
    MinSpareServers    15
    MaxSpareServers   75
    MaxClients       250
    MaxRequestsPerChild  1000
    </IfModule>

it may be my imagination, but the system appears to fail more
often with KeepAlive On.

Also, when I restart apache, I see many of these:

    [warn] child process 386 still did not exit, sending a SIGTERM

I expect these are the listener-queue and hung processes.

That's about it.  Any guesses what might be happening in there?
Any ideas how I might find out more about it?

-- 
Gary Lawrence Murphy <garym-/8Z/[email protected]> ==============================
www.teledyn.com - blog.teledyn.com - irish.teledyn.com - sbp.teledyn.com
====================== The present moment is a powerful goddess (Goethe)

___________________________________
fastcgi-developers mailing list
http://fastcgi.com/fastcgi-developers/