Re: My "incomplete headers (0 bytes)" ... [UPDATE]

We just upgraded to Red Hat Enterprise Linux 4, and found the same problem
with 1 out of every 300 or so requests returning the " FastCGI: incomplete
headers (0 bytes) received from server" error message. 

Prior to this upgrade, we did not have any issues with this. Given the
sporadic nature, it is very hard to debug and Loren, (or anyone else) who
has recommendations on how to trap/track these errors would be very
welcome!!! I don't even see a process id with the error message, so I have
no idea where to start looking.

On an unrelated issue we found that the permissions of /var/log/httpd are
set to 700 on RHEL 4.0 and needed to be set to 755. Thought this might help
someone else who was baffled with the permission denied errors.

> I found problem of last snap running on RedHat 8 and Apache 2.0.40
> (which comes with RedHat8 distribution).
> Mod_fastcgi was compiled using apxs and it was done fine.
>
> But problem is with a location of apache's logs. It's in /var/logs/httpd
> but apache access them by simlink /etc/httpd/logs. And mod_fastcgi
> can create its fastcgi and fastcgi/dynamic directory, but can't create
> server:
>
> (13)Permision denied: FastCGI: can't create server
> "/data/weby/script.fpl":
> bind() failed [/etc/httpd/logs/fastcgi/901F01943245634535463452409456]
>
> There is rxw right for all on directory /etc/httpd/logs/fastcgi/.
>
> Solution was simple for now: Logs are in /etc/httpd/logs without simlink
> and it works. But I think, there is something wrong.
>
> Is it bug or it's normal and I can't use simlinks or there is needed some
> right?

Regards,
Mike


-----Original Message-----
From: fastcgi-developers-bounces+mike=goacolyte.com-xGejAJT2w6yOE5Ap4OspEtHuzzzSOjJt@public.gmane.org
[mailto:fastcgi-developers-bounces+mike=goacolyte.com-xGejAJT2w6yOE5Ap4OspEtHuzzzSOjJt@public.gmane.org]
On Behalf Of Loren Osborn
Sent: Tuesday, January 31, 2006 12:58 PM
To: fastcgi-developers-xGejAJT2w6yOE5Ap4OspEtHuzzzSOjJt@public.gmane.org
Subject: Re: [FASTCGI] My "incomplete headers (0 bytes)" ... [UPDATE]

The following is an update to our problem with all details as we
currently understand them.  Any help, assistance, or referral to helpful
resources (compensation available) would be very much appreciated.

Again, thank you in advance,

-Loren Osborn
 Software Engineer

We have a web based Apache/Perl/DBI/mysql application running on a Red
Hat Enterprise Linux 4 server that we are in the last stages of
converting to FastCGI.  Our biggest remaining issue is a recurring
sporadic problem that our server-scripts are receiving usually a SIGTERM
(but occasionally a SIGPIPE) appearing to NOT originate from
mod_fastcgi. This results in no output to mod_fastcgi, an "incomplete
headers (0 bytes)" message in FastCGI's log file, and an "Internal
Server Error" reported by apache to our users.  This is unacceptable in
a production web application like ours. After much trial and error, and
process of elimination, we have tracked the characteristics of the
problem down with strace and other tools, and have determined that ...
that our Perl script appears to be dying from a SIGTERM after
Autoloader.pm fails to load DESTROY.al for DBI::Destroy().  I have no
definitive proof that one issue causes the other, but they do always
appear to happen together in sequence.

We initially were having problems with our scripts dying as a result of
trying to use stale database handles.  This was resolved with the code I
am attaching below, which is called at the beginning of each page
request via VerifyDbHandlesAreStillValid().  In the event that talking
to either of our database handles causes an error, we reconnect.  This
is when the reference to the old database handles are severed, and I
suspect when the GC happens that is causing us such grief.  Like I said
before, we are experiencing this only a small percent of the time, but
it is much too frequent to consider deploying this application until
this issue is resolved.






#-------------------------------
# Create database connection string, login and password variables
#-------------------------------
my $dbname         = 'dsi';
my @slave_hosts    = ('db-2.our-domain.com');
my $strConn_master = "DBI:mysql:database=$dbname;host=db-1.
our-domain.com";
my @astrConn_slaves = map { "DBI:mysql:database=$dbname;host=$_" }
@slave_hosts;

#-------------------------------
# Open the connection
#-------------------------------
my @aDbiInitParamsMaster = (
    $strConn_master,
    $username,
    $password,
    {
        PrintError => $bDebug,
        RaiseError => $bDebug
    }
);

my @aDbiInitParamsSlaves = map {
    [
        $_,
        $username,
        $password,
        {
            PrintError => $bDebug,
            RaiseError => $bDebug
        }
    ]
} @astrConn_slaves;

my $dbh_master;
my $dbh_slave;

# LSO: Borrowing O'reilly code snippet
# Function source:
http://www.unix.org.ua/orelly/perl/cookbook/ch04_18.htm
##################################################
# fisher_yates_shuffle( \@array ) : generate a random permutation # of
@array in place sub fisher_yates_shuffle {
    my $array = shift;
    my $i;
    for ( $i = @$array ; --$i ; ) {
        my $j = int rand( $i + 1 );
        next if $i == $j;
        @$array[ $i, $j ] = @$array[ $j, $i ];
    }
}
##################################################
# LSO: END OF CODE SNIPPET

sub SelectPrimaryDbHandle {
    if ($Common::ReportingMode) {
        $dbh = $dbh_slave;
    } else {
        $dbh = $dbh_master;
    }
}

sub InitDatabaseConnection {
    $dbh        = undef();
    $dbh_slave  = undef();
    $dbh_master = undef();

    eval { $dbh_master = DBI->connect(@aDbiInitParamsMaster); };
    unless ( ref $dbh_master ) {
        warn "MASTER MySQL SERVER: Appears to be down " .
Dumper($dbh_master);
        confess "$@ $DBI::err_str \n";
    }
    my @aDbiInitParamsSlavesCopy = @aDbiInitParamsSlaves;
    fisher_yates_shuffle( \@aDbiInitParamsSlavesCopy );
    until ( ref $dbh_slave ) {
        if ( scalar(@aDbiInitParamsSlavesCopy) ) {
            eval {
                $dbh_slave =
                  DBI->connect( @{ shift @aDbiInitParamsSlavesCopy } );
            };
            unless ( ref $dbh_slave ) {
                warn "SLAVE MySQL SERVER: Appears to be down "
                  . Dumper($dbh_slave);
            }
        } else {
            warn
"ALERT: Unable to connect to ANY slave database sever. Falling back to
master.";
            eval { $dbh_slave = DBI->connect(@aDbiInitParamsMaster); };
            unless ( ref $dbh_slave ) {
                warn "MASTER: " . Dumper($dbh_slave);
                confess "$@ $DBI::err_str \n";
            }
        }
    }
    SelectPrimaryDbHandle();
}

sub VerifyDbHandlesAreStillValid {
    my $bConnectionVerified = 0;
    my $bConnectionAttempts = 0;
    while ( !$bConnectionVerified ) {
        eval {

            # We don't care about the return value,
            # just if they cause a fatal error
            DLookUp( "practices", "COUNT(id)", "1", $dbh_slave,
"DONT_CATCH" );
            DLookUp( "practices", "COUNT(id)", "1", $dbh_master,
"DONT_CATCH" );

            # Don't have to check $dbh, it's just a copy
        };
        if ($@) {
            warn
"Caught a fatal error quereying the database. Trying to reconnect.";

            # Disconnect if we can... we don't care if we fail, but one
failure
            # shouldn't prevent the second attempt
            eval { $dbh_slave->disconnect(); };
            eval { $dbh_master->disconnect(); };
            InitDatabaseConnection();
            Time::HiRes::usleep(100);
        $bConnectionAttempts++;
        } else {
            $bConnectionVerified = 1;
        }
    }
    if($bConnectionAttempts > 0) {
        warn("Database connection verified after $bConnectionAttempts
reconnection attempts");
    }
}

InitDatabaseConnection();




___________________________________
fastcgi-developers mailing list
http://fastcgi.com/fastcgi-developers/



___________________________________
fastcgi-developers mailing list
http://fastcgi.com/fastcgi-developers/