Re: My "incomplete headers (0 bytes)" ... [UPDATE]

We were able to *solve* (fingers crossed) this problem on RHEL 4 by setting
the following:

FastCgiConfig -initial-env PERL_SIGNALS=unsafe

Without this line, the signals sent to fastcgi processes were not received
when they were sent. We have a process reaper that sends a SIGTERM to
processes that have grown too large so they can exit gracefully.

Before this config line was added, the signal handler did not fire until the
*next* request. The accept() loop seemed to block/suppress these signals. As
soon as the next request was received, the process promptly exited with no
content and the "incomplete headers (0 bytes)" error would show up.

We are using the standard RHEL 4 dist with a web based Apache/Perl/DBI/mysql
application. The same apps had run for a long time under RHEL 2.1 (Perl 5.6,
Apache 1.3).

Hope this helps,
Mike


-----Original Message-----
From: fastcgi-developers-bounces+mike=goacolyte.com-xGejAJT2w6yOE5Ap4OspEtHuzzzSOjJt@public.gmane.org
[mailto:fastcgi-developers-bounces+mike=goacolyte.com-xGejAJT2w6yOE5Ap4OspEtHuzzzSOjJt@public.gmane.org]
On Behalf Of Loren Osborn
Sent: Tuesday, January 31, 2006 12:58 PM
To: fastcgi-developers-xGejAJT2w6yOE5Ap4OspEtHuzzzSOjJt@public.gmane.org
Subject: Re: [FASTCGI] My "incomplete headers (0 bytes)" ... [UPDATE]

The following is an update to our problem with all details as we
currently understand them.  Any help, assistance, or referral to helpful
resources (compensation available) would be very much appreciated.

Again, thank you in advance,

-Loren Osborn
 Software Engineer

We have a web based Apache/Perl/DBI/mysql application running on a Red
Hat Enterprise Linux 4 server that we are in the last stages of
converting to FastCGI.  Our biggest remaining issue is a recurring
sporadic problem that our server-scripts are receiving usually a SIGTERM
(but occasionally a SIGPIPE) appearing to NOT originate from
mod_fastcgi. This results in no output to mod_fastcgi, an "incomplete
headers (0 bytes)" message in FastCGI's log file, and an "Internal
Server Error" reported by apache to our users.  This is unacceptable in
a production web application like ours. After much trial and error, and
process of elimination, we have tracked the characteristics of the
problem down with strace and other tools, and have determined that ...
that our Perl script appears to be dying from a SIGTERM after
Autoloader.pm fails to load DESTROY.al for DBI::Destroy().  I have no
definitive proof that one issue causes the other, but they do always
appear to happen together in sequence.

We initially were having problems with our scripts dying as a result of
trying to use stale database handles.  This was resolved with the code I
am attaching below, which is called at the beginning of each page
request via VerifyDbHandlesAreStillValid().  In the event that talking
to either of our database handles causes an error, we reconnect.  This
is when the reference to the old database handles are severed, and I
suspect when the GC happens that is causing us such grief.  Like I said
before, we are experiencing this only a small percent of the time, but
it is much too frequent to consider deploying this application until
this issue is resolved.






#-------------------------------
# Create database connection string, login and password variables
#-------------------------------
my $dbname         = 'dsi';
my @slave_hosts    = ('db-2.our-domain.com');
my $strConn_master = "DBI:mysql:database=$dbname;host=db-1.
our-domain.com";
my @astrConn_slaves = map { "DBI:mysql:database=$dbname;host=$_" }
@slave_hosts;

#-------------------------------
# Open the connection
#-------------------------------
my @aDbiInitParamsMaster = (
    $strConn_master,
    $username,
    $password,
    {
        PrintError => $bDebug,
        RaiseError => $bDebug
    }
);

my @aDbiInitParamsSlaves = map {
    [
        $_,
        $username,
        $password,
        {
            PrintError => $bDebug,
            RaiseError => $bDebug
        }
    ]
} @astrConn_slaves;

my $dbh_master;
my $dbh_slave;

# LSO: Borrowing O'reilly code snippet
# Function source:
http://www.unix.org.ua/orelly/perl/cookbook/ch04_18.htm
##################################################
# fisher_yates_shuffle( \@array ) : generate a random permutation # of
@array in place sub fisher_yates_shuffle {
    my $array = shift;
    my $i;
    for ( $i = @$array ; --$i ; ) {
        my $j = int rand( $i + 1 );
        next if $i == $j;
        @$array[ $i, $j ] = @$array[ $j, $i ];
    }
}
##################################################
# LSO: END OF CODE SNIPPET

sub SelectPrimaryDbHandle {
    if ($Common::ReportingMode) {
        $dbh = $dbh_slave;
    } else {
        $dbh = $dbh_master;
    }
}

sub InitDatabaseConnection {
    $dbh        = undef();
    $dbh_slave  = undef();
    $dbh_master = undef();

    eval { $dbh_master = DBI->connect(@aDbiInitParamsMaster); };
    unless ( ref $dbh_master ) {
        warn "MASTER MySQL SERVER: Appears to be down " .
Dumper($dbh_master);
        confess "$@ $DBI::err_str \n";
    }
    my @aDbiInitParamsSlavesCopy = @aDbiInitParamsSlaves;
    fisher_yates_shuffle( \@aDbiInitParamsSlavesCopy );
    until ( ref $dbh_slave ) {
        if ( scalar(@aDbiInitParamsSlavesCopy) ) {
            eval {
                $dbh_slave =
                  DBI->connect( @{ shift @aDbiInitParamsSlavesCopy } );
            };
            unless ( ref $dbh_slave ) {
                warn "SLAVE MySQL SERVER: Appears to be down "
                  . Dumper($dbh_slave);
            }
        } else {
            warn
"ALERT: Unable to connect to ANY slave database sever. Falling back to
master.";
            eval { $dbh_slave = DBI->connect(@aDbiInitParamsMaster); };
            unless ( ref $dbh_slave ) {
                warn "MASTER: " . Dumper($dbh_slave);
                confess "$@ $DBI::err_str \n";
            }
        }
    }
    SelectPrimaryDbHandle();
}

sub VerifyDbHandlesAreStillValid {
    my $bConnectionVerified = 0;
    my $bConnectionAttempts = 0;
    while ( !$bConnectionVerified ) {
        eval {

            # We don't care about the return value,
            # just if they cause a fatal error
            DLookUp( "practices", "COUNT(id)", "1", $dbh_slave,
"DONT_CATCH" );
            DLookUp( "practices", "COUNT(id)", "1", $dbh_master,
"DONT_CATCH" );

            # Don't have to check $dbh, it's just a copy
        };
        if ($@) {
            warn
"Caught a fatal error quereying the database. Trying to reconnect.";

            # Disconnect if we can... we don't care if we fail, but one
failure
            # shouldn't prevent the second attempt
            eval { $dbh_slave->disconnect(); };
            eval { $dbh_master->disconnect(); };
            InitDatabaseConnection();
            Time::HiRes::usleep(100);
        $bConnectionAttempts++;
        } else {
            $bConnectionVerified = 1;
        }
    }
    if($bConnectionAttempts > 0) {
        warn("Database connection verified after $bConnectionAttempts
reconnection attempts");
    }
}

InitDatabaseConnection();




___________________________________
fastcgi-developers mailing list
http://fastcgi.com/fastcgi-developers/



___________________________________
fastcgi-developers mailing list
http://fastcgi.com/fastcgi-developers/