RE: reloading perl modules under heavy load

Hi - Are you loading that at startup?  Is it necessary to reload the module while running, or can you load it once?  We had something similar that looked like random loading problems on Debian about 3 years back.  When I switched to put all of the use statements in a startup.pl and configured apache to run that at launch, the problem went away.

Jonathan L Brown
Senior Software Engineer
JOBTARGET
225 State Street, Suite 300
New London, CT 06320

Ph: (810) 938-7115
Email:[email protected]
Web: www.JobTarget.com<http://www.jobtarget.com/>

[image001]

As I usually am offsite, please call my mobile number in the event of emergency.

From: Jason Aubrey [mailto:[email protected]]
Sent: Wednesday, October 03, 2012 9:46 AM
To: [email protected]
Subject: reloading perl modules under heavy load

Hi All,

We have  modperl application running on 64bit (2 cores), freebsd 8.2, perl 1.14.1, modperl 2.0.5, apache 2.2.21, 24GB of ram and about 5.5k users. Additional system details are below.  Under heavy load we are seeing errors of the form
Failed to evaluate module Parser::Legacy: Attempt to reload Parser/Legacy.pm aborted.\nCompilation failed in require at (eval 15635) line 1....

I vaguely suspect this is a concurrency issue.  Requests that generate such errors often, but not always, call to GD.pm to build dynamically generated images.  The errors often, but not always, refer to Parser::Legacy a module which is part of our application, but I think this is a red-herring since it only occurs under load, and I've seen such errors refer to other modules, for example DateTime.

My server is the largest production freebsd machine running this application; most servers of this size run Red Hat or Ubuntu and as far as I can tell nobody has seen this error before.

I honestly have no idea if this is a modperl issue, an apache issue, a freebsd issue, etc. But I thought I would start here since this is where the rubber (requests) meet the road (perl).  Has anyone see such an error before? Or know of a similar case that might give me some additional clues?

Thanks very much,
Jason Aubrey

OS:

FreeBSD 8.2-RELEASE-p10


Perl:
modperl 2.0.5

  Platform:
    osname=freebsd, osvers=8.2-release-p2, archname=amd64-freebsd
    uname='freebsd ww64 8.2-release-p2 freebsd 8.2-release-p2 #0: sat sep 17 13:09:25 cdt 2011 root@ww64:usrobjusrsrcsysvmware amd64 '
    config_args='-sde -Dprefix=/usr/local -Darchlib=/usr/local/lib/perl5/5.14.1/mach -Dprivlib=/usr/local/lib/perl5/5.14.1 -Dman3dir=/usr/local/lib/perl5/5.14.1/perl/man/man3 -Dman1dir=/usr/local/man/man1 -Dsitearch=/usr/local/lib/perl5/site_perl/5.14.1/mach -Dsitelib=/usr/local/lib/perl5/site_perl/5.14.1 -Dscriptdir=/usr/local/bin -Dsiteman3dir=/usr/local/lib/perl5/5.14.1/man/man3 -Dsiteman1dir=/usr/local/man/man1 -Ui_malloc -Ui_iconv -Uinstallusrbinperl -Dcc=cc -Duseshrplib -Dinc_version_list=none -Dccflags=-DAPPLLIB_EXP="/usr/local/lib/perl5/5.14.1/BSDPAN" -Doptimize=-O2 -fno-strict-aliasing -pipe -march=native -Ui_gdbm -Dusethreads=n -Dusemymalloc=y -Duse64bitint -Dusesitecustomize'
    hint=recommended, useposix=true, d_sigaction=define
    useithreads=undef, usemultiplicity=undef
    useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
    use64bitint=define, use64bitall=define, uselongdouble=undef
    usemymalloc=y, bincompat5005=undef
  Compiler:
    cc='cc', ccflags ='-DAPPLLIB_EXP="/usr/local/lib/perl5/5.14.1/BSDPAN" -DHAS_FPSETMASK -DHAS_FLOATINGPOINT_H -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include',
    optimize='-O2 -fno-strict-aliasing -pipe -march=native',
    cppflags='-DAPPLLIB_EXP="/usr/local/lib/perl5/5.14.1/BSDPAN" -DHAS_FPSETMASK -DHAS_FLOATINGPOINT_H -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include'
    ccversion='', gccversion='4.2.1 20070719  [FreeBSD]', gccosandvers=''
    intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
    ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
    alignbytes=8, prototype=define
  Linker and Libraries:
    ld='cc', ldflags =' -Wl,-E  -fstack-protector -L/usr/local/lib'
    libpth=/usr/lib /usr/local/lib
    libs=-lm -lcrypt -lutil
    perllibs=-lm -lcrypt -lutil
    libc=, so=so, useshrplib=true, libperl=libperl.so
    gnulibc_version=''
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='  -Wl,-R/usr/local/lib/perl5/5.14.1/mach/CORE'
    cccdlflags='-DPIC -fPIC', lddlflags='-shared  -L/usr/local/lib -fstack-protector'


Characteristics of this binary (from libperl):
  Compile-time options: MYMALLOC PERL_DONT_CREATE_GVSV PERL_MALLOC_WRAP
                        PERL_PRESERVE_IVUV USE_64_BIT_ALL USE_64_BIT_INT
                        USE_LARGE_FILES USE_PERLIO USE_PERL_ATOF
                        USE_SITECUSTOMIZE
  Built under freebsd
  Compiled at Sep 17 2011 19:35:32

Apache:

Server version: Apache/2.2.21 (FreeBSD)
Server built:   Sep 17 2011 20:41:36
Server's Module Magic Number: 20051115:30
Server loaded:  APR 1.4.5, APR-Util 1.3.12
Compiled using: APR 1.4.5, APR-Util 1.3.12
Architecture:   64-bit
Server MPM:     Prefork
  threaded:     no
    forked:     yes (variable process count)
Server compiled with....
 -D APACHE_MPM_DIR="server/mpm/prefork"
 -D APR_HAS_SENDFILE
 -D APR_HAS_MMAP
 -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
 -D APR_USE_FLOCK_SERIALIZE
 -D APR_USE_PTHREAD_SERIALIZE
 -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
 -D APR_HAS_OTHER_CHILD
 -D AP_HAVE_RELIABLE_PIPED_LOGS
 -D DYNAMIC_MODULE_LIMIT=128
 -D HTTPD_ROOT="/usr/local"
 -D SUEXEC_BIN="/usr/local/bin/suexec"
 -D DEFAULT_PIDLOG="/var/run/httpd.pid"
 -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
 -D DEFAULT_LOCKFILE="/var/run/accept.lock"
 -D DEFAULT_ERRORLOG="logs/error_log"
 -D AP_TYPES_CONFIG_FILE="etc/apache22/mime.types"
 -D SERVER_CONFIG_FILE="etc/apache22/httpd.conf"