slow PAM auth under load

I have a server that has been running qpopper and sendmail on RH8 for 
several years.  I am having a problem where qpopper logins can take 15-20 
seconds to authorize at peak usage times.  Yet I look at the server and 
neither CPU nor memory are maxed out.  Load average might be up to 4 or 5 
when this occurs.  Also, if I try to telnet or ssh to the server while this 
is occuring, often I will see a quick prompt for username/password, but then 
the same wait for auth to complete.

I believe the server is a 1.26GHz Xeon with 2GB memory and a RAID5 array of 
3x18GB 10K SCSI drives.

This is almost certainly a PAM question not specifically a Qpopper question, 
but I'm hoping someone can point me in the right direction.  I don't see any 
configuration that would limit PAM to a certain number of logins per second 
or something like that, so I'm stumped why the auth process gets bogged down 
when the server seems to be working hard but not by any means overloaded.

I have a newer server running the same configuration on dual 2.4GHz Xeons 
with a SATA drive and I'm not seeing the same issue, but that may be due to 
different users rather than different server.