seg fault in modperl_perl_global_request_save

1. Problem Description:

Our production apache22/mod_perl2 web server segfaults regularly. I
have tracked down this problem as much as I can, and could use some
help. This is running on FreeBSD; the same problem occurs on FreeBSD
7.0, 7.2, and 8.0. apache22 is installed from ports, with WITH_DEBUG
and WITH_EXECPTION_HOOK; mod_perl2 is installed from ports, my
attempts to compile it with debug flags appear so far to be
unsuccessful. (detailed version info below)

For no reason I can fathom, I can not seem to get apache to dump core
when it seg faults. I've set CoreDumpDirectory, I've set ulimit
coredumpsize, nothing.  No matter; I was able to install
mod_whatkilledus and mod_backtrace.

mod_whatkilledus shows that the segfaults happen on a wide variety of
requests; most requests to our main web server are handled by
mod_perl, either with a ResponseHandler or a FilterHandler.

mod_backtrace shows that the seg fault is always happening in
modperl_perl_global_request_save:

[Wed Dec 30 15:05:41 2009] pid 6955 mod_backtrace backtrace for sig 11 (thread "pid" 6955)
[Wed Dec 30 15:05:41 2009] pid 6955 mod_backtrace main() is at 8063880
0x8085a26 <ap_run_fatal_exception+70> at /usr/local/sbin/httpd
0x808752e <ap_mpm_set_exception_hook+238> at /usr/local/sbin/httpd
0x8087569 <ap_mpm_set_exception_hook+297> at /usr/local/sbin/httpd
0xbfbfffb4
0x287a83c5 <modperl_perl_global_request_save+213> at /usr/local/libexec/apache22/mod_perl.so
0x2879982d <modperl_response_handler_cgi+141> at /usr/local/libexec/apache22/mod_perl.so
0x807bc66 <ap_run_handler+70> at /usr/local/sbin/httpd
...

So, any general pointers for what direction to go next would be
appreciated. I would like to have mod_backtrace tell me what in my
perl code is triggering the error, in which case I could work around
it. Should I continue to pursue recompiling mod_perl2 and possibly
perl itself with more debugging turned on?
(http://perl.apache.org/docs/1.0/guide/help.html)

2. Used Components and their Configuration:

*** mod_perl version 2.000004

*** using /usr/local/lib/perl5/site_perl/5.8.9/mach/Apache2/BuildConfig.pm

*** Makefile.PL options:
   MP_APR_LIB     => aprext
   MP_APXS        => /usr/local/sbin/apxs
   MP_COMPAT_1X   => 1
   MP_GENERATE_XS => 1
   MP_LIBNAME     => mod_perl
   MP_USE_DSO     => 1


*** The httpd binary was not found

# I ran it by hand instead:

bsdtest# /usr/local/sbin/httpd -V
Server version: Apache/2.2.14 (FreeBSD)
Server built:   Dec 30 2009 09:46:16
Server's Module Magic Number: 20051115:23
Server loaded:  APR 1.3.8, APR-Util 1.3.9
Compiled using: APR 1.3.8, APR-Util 1.3.9
Architecture:   32-bit
Server MPM:     Prefork
   threaded:     no
     forked:     yes (variable process count)
Server compiled with....
  -D APACHE_MPM_DIR="server/mpm/prefork"
  -D APR_HAS_SENDFILE
  -D APR_HAS_MMAP
  -D APR_USE_FLOCK_SERIALIZE
  -D APR_USE_PTHREAD_SERIALIZE
  -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
  -D APR_HAS_OTHER_CHILD
  -D AP_HAVE_RELIABLE_PIPED_LOGS
  -D DYNAMIC_MODULE_LIMIT=128
  -D HTTPD_ROOT="/usr/local"
  -D SUEXEC_BIN="/usr/local/bin/suexec"
  -D DEFAULT_PIDLOG="/var/run/httpd.pid"
  -D DEFAULT_SCOREBOARD="/var/run/apache_runtime_status"
  -D DEFAULT_LOCKFILE="/var/run/accept.lock"
  -D DEFAULT_ERRORLOG="/var/log/httpd-error.log"
  -D AP_TYPES_CONFIG_FILE="etc/apache22/mime.types"
  -D SERVER_CONFIG_FILE="etc/apache22/httpd.conf"

[Wed Dec 30 15:49:00 2009] [notice] Apache/2.2.14 (FreeBSD) mod_ssl/2.2.14 OpenSSL/0.9.8k mod_apreq2-20090110/2.7.1 mod_perl/2.0.4 Perl/v5.8.9 configured -- resuming normal operations

*** (apr|apu)-config linking info

  -L/usr/local/lib -laprutil-1     -lexpat -liconv -L/usr/local/lib
  -L/usr/local/lib -lapr-1 -lcrypt  -pthread



*** /usr/local/bin/perl -V
Summary of my perl5 (revision 5 version 8 subversion 9) configuration:
   Platform:
     osname=freebsd, osvers=8.0-release, archname=i386-freebsd-64int
     uname='freebsd freebsd.org 8.0-release freebsd 8.0-release #0: fri sep 18 03:01:27 pdt 2009 [email protected]:usrsrcsysmagickernelpath i386 '
     config_args='-sde -Dprefix=/usr/local -Darchlib=/usr/local/lib/perl5/5.8.9/mach -Dprivlib=/usr/local/lib/perl5/5.8.9 -Dman3dir=/usr/local/lib/perl5/5.8.9/perl/man/man3 -Dman1dir=/usr/local/man/man1 -Dsitearch=/usr/local/lib/perl5/site_perl/5.8.9/mach -Dsitelib=/usr/local/lib/perl5/site_perl/5.8.9 -Dscriptdir=/usr/local/bin -Dsiteman3dir=/usr/local/lib/perl5/5.8.9/man/man3 -Dsiteman1dir=/usr/local/man/man1 -Ui_malloc -Ui_iconv -Uinstallusrbinperl -Dcc=cc -Duseshrplib -Dinc_version_list=none -Dccflags=-DAPPLLIB_EXP="/usr/local/lib/perl5/5.8.9/BSDPAN" -Doptimize=-O2 -pipe -fno-strict-aliasing -Ud_dosuid -Ui_gdbm -Dusethreads=n -Dusemymalloc=y -Duse64bitint'
     hint=recommended, useposix=true, d_sigaction=define
     usethreads=undef use5005threads=undef useithreads=undef usemultiplicity=undef
     useperlio=define d_sfio=undef uselargefiles=define usesocks=undef
     use64bitint=define use64bitall=undef uselongdouble=undef
     usemymalloc=y, bincompat5005=undef
   Compiler:
     cc='cc', ccflags ='-DAPPLLIB_EXP="/usr/local/lib/perl5/5.8.9/BSDPAN" -DHAS_FPSETMASK -DHAS_FLOATINGPOINT_H -fno-strict-aliasing -pipe -I/usr/local/include',
     optimize='-O2 -pipe -fno-strict-aliasing',
     cppflags='-DAPPLLIB_EXP="/usr/local/lib/perl5/5.8.9/BSDPAN" -DHAS_FPSETMASK -DHAS_FLOATINGPOINT_H -fno-strict-aliasing -pipe -I/usr/local/include'
     ccversion='', gccversion='4.2.1 20070719  [FreeBSD]', gccosandvers=''
     intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=12345678
     d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12
     ivtype='long long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
     alignbytes=4, prototype=define
   Linker and Libraries:
     ld='cc', ldflags =' -Wl,-E  -L/usr/local/lib'
     libpth=/usr/lib /usr/local/lib
     libs=-lm -lcrypt -lutil
     perllibs=-lm -lcrypt -lutil
     libc=, so=so, useshrplib=true, libperl=libperl.so
     gnulibc_version=''
   Dynamic Linking:
     dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='  -Wl,-R/usr/local/lib/perl5/5.8.9/mach/CORE'
     cccdlflags='-DPIC -fPIC', lddlflags='-shared  -L/usr/local/lib'


Characteristics of this binary (from libperl):
   Compile-time options: MYMALLOC PERL_MALLOC_WRAP USE_64_BIT_INT
                         USE_FAST_STDIO USE_LARGE_FILES USE_PERLIO
   Locally applied patches:
 	defined-or
   Built under freebsd
   Compiled at Sep 18 2009 10:03:02
   %ENV:
     PERL_LWP_USE_HTTP_10="1"
   @INC:
     /usr/local/lib/perl5/5.8.9/BSDPAN
     /usr/local/lib/perl5/site_perl/5.8.9/mach
     /usr/local/lib/perl5/site_perl/5.8.9
     /usr/local/lib/perl5/5.8.9/mach
     /usr/local/lib/perl5/5.8.9
     .

*** Packages of interest status:

Apache2            : -
Apache2::Request   : 2.12
CGI                : 3.42
ExtUtils::MakeMaker: 6.48, undef
LWP                : 5.834
mod_perl           : -
mod_perl2          : 2.000004


3. This is the core dump trace: (if you get a core dump):

   [CORE TRACE COMES HERE]

This report was generated by /usr/local/bin/mp2bug on Wed Dec 30 20:46:25 2009 GMT.


   - Alex Aminoff
     BaseSpace.net
     National Bureau of Economic Research (nber.org)