[Pound Mailing List] Weird lockup / stall on OpenBSD 5.2 w/ pound 2.7a

Hi folks - I've had a bit of an unexpected turn after upgrading one of 
my load balancers from OpenBSD 5.0 to 5.2 today.  Previously this 
machine ran pound beautifully, so I'm surprised about the change. 
Here's my setup:

- OpenBSD 5.2/i386, a very vanilla install
- OpenSSL 1.0.1e built with:

./config --prefix=/usr/local/openssl-1.0.1e 
--openssldir=/usr/local/openssl-1.0.1e threads zlib shared

- ldconfig adjustmented appropriately via

shlib_dirs="/usr/local/lib /usr/local/openssl/lib"

- Pound 2.7a built from source with these configure flags:

    --prefix=/usr/local --with-ssl=/usr/local/openssl --enable-cert1l

Yes, I verified that pound is using the right openssl lib:

# ldd /usr/local/sbin/pound
/usr/local/sbin/pound:
         Start    End      Type Open Ref GrpRef Name
         1c000000 3c007000 exe  1    0   0      /usr/local/sbin/pound
         09b5c000 29b60000 rlib 0    1   0 
/usr/local/lib/libpcreposix.so.1.5
         01018000 21025000 rlib 0    1   0 
/usr/local/openssl/lib/libssl.so.1.0.0
         0e54b000 2e58a000 rlib 0    2   0 
/usr/local/openssl/lib/libcrypto.so.1.0.0
         0b978000 2b981000 rlib 0    1   0      /usr/lib/libm.so.7.0
         0c5ac000 2c5b1000 rlib 0    2   0      /usr/lib/libpthread.so.16.0
         0d70a000 2d738000 rlib 0    1   0      /usr/lib/libc.so.65.0
         07de8000 27dfc000 rlib 0    1   0 
/usr/local/lib/libpcre.so.3.0
         07d20000 27d27000 rlib 0    2   0      /usr/lib/libz.so.4.1
         048c7000 048c7000 rtld 0    1   0      /usr/libexec/ld.so


Pound is configured w/ ~80 listen directives, split HTTP and HTTPS. 
Other config bits:

User        "proxy"
Group       "proxy"
RootJail    "/var/pound"
Alive       3
TimeOut     120
LogLevel    5
Daemon      1
Control     "/tmp/pound.sock"
DynScale    0
ConnTO      5
Threads     128

I ran some load against the machine via ab, hitting the pound on 
127.0.0.1.  No problems.  When I switch some real traffic to the box (by 
promoting the carp interface to master), pound seems to get stuck pretty 
quickly and requests start to stall.  If I look in top, pound is in 
state "fdlock", and 'netstat -na' shows a bunch of nonzero entries in 
the recv-q fields.  CPU load is near-zero.  And poundctl shows 0 queued 
requests, though this command takes a while to return.

So it looks like something has deadlocked and pound isn't processing 
requests.  Eventually requests do appear to get processed, but extremely 
slowly, and pound returns to the 'wait' state in top.  If I then do 
another request on localhost, it hangs again.

Using ktrace/kdump, I narrowed things down somewhat.  The slowdown 
appears to happen on the socket() call in this sequence:

  15891/1004731 pound    1363487694.436047 GIO   fd 1 read 114 bytes
... HTTP REQUEST SNIPPED HERE ...
  15891/1004731 pound    1363487694.436055 RET   read 114/0x72
  15891/1004731 pound    1363487694.436086 CALL 
gettimeofday(0x81a18b98,0x81a18b90)
  15891/1004731 pound    1363487694.436096 STRU  struct timeval { "Mar 
16 22:34:54 2013".436093 }
  15891/1004731 pound    1363487694.436103 RET   gettimeofday 0
  15891/1004731 pound    1363487694.436112 CALL  gettimeofday(0x81a18b94,0)
  15891/1004731 pound    1363487694.436121 STRU  struct timeval { "Mar 
16 22:34:54 2013".436119 }
  15891/1004731 pound    1363487694.436127 RET   gettimeofday 0
  15891/1004731 pound    1363487694.436290 CALL 
socket(PF_INET,SOCK_STREAM,IPPROTO_IP)

... 80 seconds go by...

  15891/1004731 pound    1363487774.413972 RET   socket 2
  15891/1004731 pound    1363487774.413982 CALL 
fcntl(0x2,F_GETFL,0x74be3f00)
  15891/1004731 pound    1363487774.413990 RET   fcntl 2
  15891/1004731 pound    1363487774.413996 CALL 
fcntl(0x2,F_SETFL,0x6<O_RDWR|O_NONBLOCK>)
  15891/1004731 pound    1363487774.414003 RET   fcntl 0
  15891/1004731 pound    1363487774.414010 CALL 
connect(0x2,0x7d8d35f0,0x10)

Does anybody have pound working on OpenBSD 5.2?  Has anyone encountered 
this problem or anything similar?

-- 
Dave Steinberg
http://www.geekisp.com/
http://www.steinbergcomputing.com/
http://www.redterror.net/

--
To unsubscribe send an email with subject unsubscribe to [email protected]
Please contact [email protected] for questions.