[Opensymphony-oscache] Cache and clustering (again) on 2.1RC1

For some reason I can't seem to post to the mailinglist so here goes:
(Pasted from my mail I sent to the list)

I know that this has been discussed before (and I have read most of the entries) but I still have problems understanding how things should work when deployed in a clustered config.

I've got OSCache configured for clustered mode with the default settings for the multicast config:

cache.cluster.properties=UDP(mcast_addr=239.200.6.223;mcast_port=9831;ip_ttl=32;mcast_send_buf_size=150000;mcast_recv_buf_size=80000):PING(timeout=2000;num_initial_members=3):MERGE2(min_interval=5000;max_interval=10000):FD_SOCK:VERIFY_SUSPECT(timeout=1500):pbcast.NAKACK(gc_lag=50;retransmit_timeout=300,600,1200,2400,4800):pbcast.STABLE(desired_avg_gossip=20000):UNICAST(timeout=5000):FRAG(frag_size=8096;down_thread=false;up_thread=false):pbcast.GMS(join_timeout=5000;join_retry_timeout=2000;shun=false;print_local_addr=true)

I'm running on two clustered Weblogic8.1SP3 instances and when deploying and running them I get (in both logs):

[04/29/05 12:33:42.219 INFO ] JavaGroups clustering support started successfully (JavaGroupsBroadcastingListener.java:99) in thread "ExecuteThread: '13' for queue: 'weblogic.kernel.Default'"

In both logs at some point. I also get a bunch of these: (example from my second instance log):

[04/29/05 12:33:42.228 INFO ] A new member at address 'lonlxwebhost10:1199' has joined the cluster (JavaGroupsBroadcastingListener.java:166) in thread "PullPushAdapterThread" [04/29/05 12:33:42.230 INFO ] A new member at address 'lonlxwebhost3:52846' has joined the cluster (JavaGroupsBroadcastingListener.java:166) in thread "PullPushAdapterThread" [04/29/05 12:33:42.231 INFO ] A new member at address 'lonlxwebhost3:59573' has joined the cluster (JavaGroupsBroadcastingListener.java:166) in thread "PullPushAdapterThread" [04/29/05 12:33:42.231 INFO ] A new member at address 'lonlxwebhost3:59583' has joined the cluster (JavaGroupsBroadcastingListener.java:166) in thread "PullPushAdapterThread" [04/29/05 12:33:42.231 INFO ] A new member at address 'lonlxwebhost10:28388' has joined the cluster (JavaGroupsBroadcastingListener.java:166) in thread "PullPushAdapterThread"

So they seem to be seeing eachother even though I don't really understand why so many "members" need to join the cluster since I only have two WLS instances in the cluster. Are there several threads per instance that listens?

So far everything seems well and fine. So what I do is that I turn the refresh value down to some low number (30 secs) when getting from the cache so I will get stale hits when accessing the cache on one instance (first filling the cache with some data obviosly. So if I haven't got this totally wrong this should broadcast some kind of "NeedsRefresh" message to the other instance for all the objects that gets a stale hit? No? If not the other instance would now know that the data it contains is stale and use it. The only thing I can think of now is that the data needs to be on both hosts for the broadcast of the stale hit to happen?

I get nothing in the other instance log. Absolutely nothing.

But! When I flush the cache on my first instance I get this in the sender log:

[04/29/05 12:44:31.500 DEBUG] cacheFushed called (com.opensymphony.oscache.base.events.CachewideEvent@559bd12) (AbstractBroadcastingListener.java:81) in thread "ExecuteThread: '10' for queue: 'weblogic.kernel.Default'"

And this in the reciever log:

[04/29/05 12:44:31.539 INFO ] Cluster notification (type=4, data=Fri Apr 29 12:44:31 BST 2005) was received. (AbstractBroadcastingListener.java:148) in thread "PullPushAdapterThread" [04/29/05 12:44:31.550 DEBUG] ---------------------> CacheFlushed! (myCacheEntryEventListener.java:150) in thread "PullPushAdapterThread"

And my cache on the other instance actually get flushed!

So it seems to work for flushes but not for broadcasting refreshes after stale hits. Is the above behaviour expected?

The problem we have been having is that we ocationally get stale data from the day before (!) even if our refresh value is one hour in Production. This only seems to happen when one intance locks up and the requests need to go to the other instance. This would definitely suggest that the "Need refresh" notification does not reach it's destination....

Help please?

/Jonas Larsson

PS. Looking closer in my logs I also get this *After* my cache has succesfully flushed on the primary instance (and sent the successful broadcast described above):

[04/29/05 12:44:31.519 ERROR] exception=java.lang.IllegalArgumentException: java.lang.ClassNotFoundException: org.jgroups.blocks.NotificationBus$Info (NotificationBus.java:286) in thread "PullPushAdapterThread"

Should I be worried about this? I seem to remember that I have updated to a more recent version of Jgroups to fix the "changed package name problem" that plagued the 2.0.2 version Or that was fixed in the 2.1RC1 version I am currently using so that should not really be the problem....

---------------------------------------------------------------------
Posted via Jive Forums
http://forums.opensymphony.com/thread.jspa?threadID=1875&messageID=5786#5786