Re: reloading perl modules under heavy load

On Thu, Oct 4, 2012 at 9:53 AM, demerphq <[email protected]> wrote:

> On 4 October 2012 16:36, Jason Aubrey <[email protected]> wrote:
> > Thanks all for your replies to my question.
> >
> > Because of the nature of our application, we can't really load
> everything at
> > start up, but I did some digging and there are clearly some
> inefficiencies
> > here and the situation would indeed be improved by cleaning these up.
> > However, it does look like it may be the interaction between these
> > inefficiencies and freebsd in particular that is causing the error.
> >
> > So, here's what I did: on a linux development box I wrote a script using
> > Linux::Inotify2 to watch the directories containing the code for our
> > application that we (think we) have to load at run time.  A typical
> request
> > generated an average of 225 IN_OPEN inotify events.  (FWIW, it also
> > generated around 850 IN_ACCESS events.  I don't understand the
> difference,
> > but IN_OPEN events seem more relevant to what I'm getting to.)
> >
> > Now, we have MaxClients set super high, but with just 150 concurrent
> > requests that's 33,750 open files and that's not taking into account
> files
> > opened for other processes on the machine, such as the kernel or the web
> > server, etc.
> >
> > But, here's where freebsd comes in.  The freebsd kernel has a bunch of
> > tunables, one of which is kern.maxfiles. The freebsd manual says "This
> > variable indicates the maximum number of file descriptors on your
> system...
> > Each open file, socket, or fifo uses one file descriptor. A large-scale
> > production server may easily require many thousands of file descriptors,
> > depending on the kind and number of services running concurrently"
> >
> > Well, we had kern.maxfiles = 12328.  Also possibly relevant is
> > kern.ipc.somaxconn which "limits the size of the listen queue for
> accepting
> > new TCP connections".  We had that at the default of 128.  I also don't
> > understand the interaction between this setting and apache's MaxClients
> > since when we had kern.ipc.somaxconn = 128 and MaxClients 150 we quickly
> > maxed out max clients.
> >
> > So, we bumped up kern.maxfiles to 50000 and kern.ipc.somaxconn to 8192
> and
> > things appeared to be better last night under heavy load. But, as I said,
> > there are clearly some in efficiencies too in the way our application
> loads
> > code at run time, and cleaning these up would certainly help.
>
> You would also see reduced memory pressure and performance
> improvements. Perl is actually pretty slow at compiling code (in
> comparison to executing code), and code that is compiled prefork is
> going to be shared, whereas code compiled post-fork is going to be
> duplicated per process.
>
> By preloading you will both win performance improvements and reduce
> memory pressure on your application.
>
> I strongly recommend you approach this subject aggressively.
>
> yves
>

Hi all,

Here's another update which you will be glad to hear because it proves you
are awesome (or at least that you were right).  The freebsd tuning I
mentioned below helped but didn't fix the problem.

We still got a bunch of such errors after this tuning (fewer, but still a
lot).  Well, I added just one file to the list of code compiled in at start
up and since doing so we have had exactly zero such errors.  Admittedly,
that file pulls in a large package, but this single 8-character change to
one line of code seems to have had a tremendously positive impact on the
performance of our application.

Thanks again for all of of your help,
Jason