stall in java (affects WO and JBoss apps)

Hi

I have a Linux server that has developed a bad habit of stalling in  
java at the end of startup before an app starts running.  In WO,  
monitor launches the instance, and it goes through to here

.
.
.
[2004-02-02 16:14:21 MST] <main> WOAdditionalAdaptors=()
[2004-02-02 16:14:21 MST] <main> WOLifebeatDestinationPort=1085
[2004-02-02 16:14:21 MST] <main> WOPort=2001
[2004-02-02 16:14:21 MST] <main> WOLifebeatInterval=30
[2004-02-02 16:14:21 MST] <main> WOSocketCacheSize=100
[2004-02-02 16:14:21 MST] <main>  
WOUserDirectory=/usr/local/webobjects/wo52/Library/WebObjects/ 
JavaApplications/wotaskd.woa
[2004-02-02 16:14:21 MST] <main> WODisplayExceptionPages=false
[2004-02-02 16:14:21 MST] <main> com.webobjects.pid = 3424
[2004-02-02 16:14:21 MST] <main> ------------------------

then hangs for 3.xx minutes and then continues on with the initial db  
fetches etc.  Monitor thinks it is dead though since it doesn't respond  
back within that 3.xx minutes.  I also have a JBoss app that goes  
through all its inits and stalls for 3.xx minutes before it starts  
running.  So I don't think it is a WO problem per se, but a java  
installation problem.  I have not run WO on this server before, but the  
JBoss app was working previously.  Not a lot  has changed on it either  
:-(  The jvm installed is both the Sun 1.4.1_03 and 1.4.2_03 (it  
happens with both).  During the pause time, I sent a SIGQUIT to the  
main thread of the WO app and this is what it looks like:  Can anyone  
see anything in this dump on where it is waiting? I think the "main"  
but I don't know what it is really trying to do there.  I have checked  
and at least from the command line I can do DNS lookups and reverse  
lookups so it doesn't appear to be waiting for a DNS. It kind of looks  
like a DNS request is sent about the stall time by looking at the  
firewall logs, but it is hard to see and looking at debug output from  
my DNS server on another host I don't see a DNS request coming in that  
stalls.


Full thread dump Java HotSpot(TM) Client VM (1.4.1_03-b02 mixed mode):

"Signal Dispatcher" daemon prio=1 tid=0x0x4c404f28 nid=0xe2e waiting on  
condition [0..0]

"Finalizer" daemon prio=1 tid=0x0x4c400578 nid=0xe2b in Object.wait()  
[4c3e2000..4c3e28b0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0x446f2948> (a java.lang.ref.ReferenceQueue$Lock)
         at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:111)
         - locked <0x446f2948> (a java.lang.ref.ReferenceQueue$Lock)
         at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:127)
         at  
java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

"Reference Handler" daemon prio=1 tid=0x0x8086748 nid=0xe2a in  
Object.wait() [4c361000..4c3618b0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0x446f29b0> (a java.lang.ref.Reference$Lock)
         at java.lang.Object.wait(Object.java:426)
         at  
java.lang.ref.Reference$ReferenceHandler.run(Reference.java:113)
         - locked <0x446f29b0> (a java.lang.ref.Reference$Lock)

"main" prio=1 tid=0x0x80528e0 nid=0xe27 runnable [bfffb000..bfffd0dc]
         at java.net.PlainSocketImpl.initProto(Native Method)
         at java.net.PlainSocketImpl.<clinit>(PlainSocketImpl.java:62)
         at java.net.ServerSocket.setImpl(ServerSocket.java:236)
         at java.net.ServerSocket.<init>(ServerSocket.java:178)
         at  
com.webobjects.appserver._private.WODefaultAdaptor._initServerSocket(WOD 
efaultAdaptor.java:157)
         at  
com.webobjects.appserver._private.WODefaultAdaptor.<init>(WODefaultAdapt 
or.java:140)
         at  
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
         at  
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorA 
ccessorImpl.java:39)
         at  
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingCons 
tructorAccessorImpl.java:27)
         at  
java.lang.reflect.Constructor.newInstance(Constructor.java:274)
         at  
com.webobjects.foundation._NSUtilities.instantiateObject(_NSUtilities.ja 
va:529)
         at  
com.webobjects.appserver.WOApplication.adaptorWithName(WOApplication.jav 
a:909)
         at  
com.webobjects.appserver.WOApplication._initAdaptors(WOApplication.java: 
955)
         at  
com.webobjects.appserver.WOApplication.<init>(WOApplication.java:509)
         at Application.<init>(Application.java:30)
         at  
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
         at  
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorA 
ccessorImpl.java:39)
         at  
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingCons 
tructorAccessorImpl.java:27)
         at  
java.lang.reflect.Constructor.newInstance(Constructor.java:274)
         at java.lang.Class.newInstance0(Class.java:306)
         at java.lang.Class.newInstance(Class.java:259)
         at  
com.webobjects.appserver.WOApplication.main(WOApplication.java:296)
         at Application.main(Application.java:26)
         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
         at  
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav 
a:39)
         at  
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor 
Impl.java:25)
         at java.lang.reflect.Method.invoke(Method.java:324)
         at  
com.webobjects._bootstrap.WOBootstrap.main(WOBootstrap.java:71)

"VM Thread" prio=1 tid=0x0x8083518 nid=0xe29 runnable

"VM Periodic Task Thread" prio=1 tid=0x0x4c403c68 nid=0xe2c waiting on  
condition
"Suspend Checker Thread" prio=1 tid=0x0x4c404560 nid=0xe2d runnable