[jetty-user] Hanging threads when using Jetty 6 with IE8 & IE9

Greetings,


Jetty is great, we love it so much, you are all doing such a wonderful work
!



We are experiencing a high priority problem at one of our customer sites.

The symptoms we are seeing are, that simple browser GET requests for images
or static resources just hang for up 50 seconds with no apparent reason.



We are using:

Jetty 6.x (latest) with BIO connector running on Windows 2008 32 bit with
Java 1.6.x (latest)



We are using the following thread pool settings in jetty.xml:

    <Set name="ThreadPool">

      <New class="org.mortbay.thread.QueuedThreadPool">

        <Set name="minThreads">10</Set>

        <Set name="maxThreads">200</Set>

        <Set name="lowThreads">20</Set>

        <Set name="SpawnOrShrinkAt">2</Set>

      </New>

    </Set>



On the client side the problem only occurs when using IE8 and IE9 but not
with IE7 (probably since IE7 is limited to two concurrent requests but we
are not sure)



When the problem occurs we can see in the server logs the following events:

1.       A new web page is sent is to the browser as a response to a client
request.

2.       The client browser then submits 10 concurrent requests at once,
for loading static resources such as images, JavaScripts, CSS etc.

3.       Jetty dispatches all 10 threads in its thread pool to serve these
resource.

4.       All Jetty threads are now hang, with the following stack trace
waiting for something:

"9543875@qtp-23978087-3" prio=6 tid=0x07b07c00 nid=0x1f6c runnable
[0x0908f000..0x0908fbe8]

   java.lang.Thread.State: RUNNABLE

        at java.net.SocketInputStream.socketRead0(Native Method)

        at java.net.SocketInputStream.read(SocketInputStream.java:129)

        at org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:382)

        at org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:107)

        at
org.mortbay.jetty.bio.SocketConnector$Connection.fill(SocketConnector.java:198)

        at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:290)

        at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)

        at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)

        at
org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)

        at
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:520)



5.       At some point around 50 seconds later we receive the following
exception in the log:

java.net.SocketTimeoutException: Read timed out
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:150)
        at java.net.SocketInputStream.read(SocketInputStream.java:121)
        at org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:382)
        at org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:107)
        at
org.mortbay.jetty.bio.SocketConnector$Connection.fill(SocketConnector.java:198)
        at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:290)
        at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
        at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
        at
org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)



6.       After this exception the Jetty threads are operational again
freeing the client browser to finish rendering the page.

 We are able to reproduce this problem fairly easily in our test
environment even with a single session.


We have two questions:

1.       Why are the Jetty threads hanging on socket read after serving a
static resources ?

2.       Why doesn’t the Jetty thread pool spawn more threads up to 200
when all initial 10 threads are locked ?


We already made the following observation:

1.       If we change SpawnOrShrinkAt to 0 then the thread pool is able to
spawn more threads and this improves the situation and delays the
occurrence of the problem.



Any advice on this would be much appreciated,



Best regards,

Lior