Re: JCS remote cache client shutdown behaviour

I don't think that shutdown will properly kill the event queues on the serv=
er side.  The server will queue and retry 10 times before killing the queue=
 and marking it non functional.  If you have a large number of items and lo=
ts of clients that keep going down, then you could see a memory spike.  =0A=
=0AWe should look into the client shutdown process.  =0A=0AAaron=0A=0A--- O=
n Mon, 9/7/09, Niall Gallagher <[email protected]> wrote:=0A=0A> From: N=
iall Gallagher <[email protected]>=0A> Subject: JCS remote cache client =
shutdown behaviour=0A> To: "JCS Developers List" <[email protected]=
g>=0A> Date: Monday, September 7, 2009, 7:50 AM=0A> Hi,=0A> =0A> I'm wonder=
ing if anyone can explain the sequence of steps=0A> the JCS client=0A> code=
 is supposed to follow when=0A> CompositeCacheManager.shutDown() is=0A> cal=
led client-side? We are intermittently seeing high=0A> memory usage in=0A> =
our JCS remote server, which appears to be caused by large=0A> backlogs of=
=0A> event objects queued for delivery to client machines which=0A> have be=
en=0A> shut down, even though we are shutting down our client=0A> machines=
=0A> gracefully using the method above. This is certainly=0A> aggravated by=
 our=0A> network's architecture, but I'm not sure if the root cause=0A> mig=
ht be a=0A> bug in JCS or I'm not understanding what should happen=0A> prop=
erly.=0A> =0A> When we call CompositeCacheManager.shutDown() on a client=0A=
> machine, from=0A> our client-side logs it appears that the dispose() meth=
od=0A> in this object=0A> is getting called correctly for each cache region=
:=0A> http://svn.apache.org/viewvc/jakarta/jcs/trunk/src/java/org/apache/jc=
s/auxiliary/remote/RemoteCacheListener.java?view=3Dmarkup=0A> =0A> However =
that method appears to just unexport the RMI=0A> RemoteCacheListener=0A> ob=
ject for each region client-side; basically terminating=0A> the=0A> client-=
side end of the event delivery connection. Before=0A> disconnecting=0A> tho=
ugh, shouldn't this method notify the server that the=0A> client is about=
=0A> to disconnect?=0A> =0A> Subsequently we often see errors like this in =
the remote=0A> server log:=0A> =0A> =0A> 07-Sep 13:52:13,347 INFO=A0 [jcs.e=
ngine.CacheEventQueue]=0A> Error while running event from Queue: RemoveEven=
t for [GAN:=0A> groupId=3D[groupId=3D<region name>, defaultGroup],=0A> attr=
Name=3D<cache key>]. Retrying...=0A> 07-Sep 13:52:13,747 WARN=A0 [jcs.engin=
e.CacheEventQueue]=0A> java.rmi.ConnectException: Connection refused to hos=
t:=0A> <client machine ip address>; nested exception is:=0A> =A0 =A0 =A0 =
=A0 java.net.ConnectException:=0A> Connection refused=0A> 07-Sep 13:52:13,7=
48 WARN=A0 [jcs.engine.CacheEventQueue]=0A> Error while running event from =
Queue: RemoveEvent for [GAN:=0A> groupId=3D[groupId=3D<region name>, defaul=
tGroup],=0A> attrName=3D<cache key>]. Dropping Event and marking=0A> Event =
Queue as non-functional.=0A> =0A> =0A> ...this implies the remote server co=
ntinues to try to=0A> deliver events to=0A> the JCS client which disconnect=
ed, as if the client didn't=0A> de-register=0A> itself before disconnecting=
.=0A> =0A> Perhaps I've missed something in the code.=0A> =0A> I see that t=
he RemoteCacheServer API (to which clients=0A> connect) does in=0A> fact ha=
ve a server-side dispose() method which (on initial=0A> investigation) woul=
d "de-register" the client from the=0A> server's list of=0A> event listener=
s. Could it be that JCS clients are simply=0A> not calling=0A> this method?=
..=0A> http://svn.apache.org/viewvc/jakarta/jcs/trunk/src/java/org/apache/j=
cs/auxiliary/remote/server/RemoteCacheServer.java?view=3Dmarkup=0A> =0A> =
=0A> This issue is a problem for us depending on which network=0A> subnet t=
he=0A> client machine is in. Basically our network is divided into=0A> 2 su=
bnets,=0A> with a fairly rubbish (or overly-strict) router/firewall=0A> bet=
ween the two=0A> subnets. This router does not relay networking errors (ICM=
P=0A> error=0A> messages) between the two subnets. When a machine in one=0A=
> subnet goes=0A> offline and a machine in the other subnet tries to connec=
t=0A> to it, our=0A> router does not notify the source machine that the tar=
get=0A> machine is=0A> offline, and so the source machine waits indefinitel=
y (i.e.=0A> with a=0A> socket in the open wait state) for a response from t=
he=0A> target machine.=0A> On the other hand when both machines are in the =
same=0A> subnet, the source=0A> machine gets a "host not reachable" excepti=
on immediately=0A> when a target=0A> machine is offline.=0A> =0A> Anyway...=
 the problem is when we shut down a client machine=0A> in a=0A> different s=
ubnet, the JCS remote server builds up a large=0A> backlog of=0A> cache eve=
nt objects, presumably trying to connect to a=0A> disconnected=0A> client, =
and eventually runs out of memory. We determine=0A> this using the=0A> JDK'=
s jmap command - we find a large number of PutEvent and=0A> RemoveEvent=0A>=
 objects in the remote server's memory. We don't have the=0A> issue when bo=
th=0A> machines are in the same subnet, but I wonder if that's=0A> because =
JCS=0A> remote server is relying on the networking errors, and is=0A> de-re=
gistering=0A> clients automatically after a certain number of failed=0A> at=
tempts to=0A> connect to the client. i.e. perhaps clients are not=0A> de-re=
gistering=0A> themselves gracefully from the remote server in the first=0A>=
 place.=0A> =0A> Does anyone have any experience with this- anyone regularl=
y=0A> see "Error=0A> while running event from Queue" in the remote server l=
ogs?=0A> I realise our=0A> network setup is partly to blame here, but perha=
ps the root=0A> cause is=0A> that client's are not de-registering properly.=
=0A> =0A> Many thanks in advance,=0A> =0A> Niall=0A>