Re: [jgroups-dev] Advice on best practices for implementing a new protocol

Mike Jensen wrote:
> I started work today to implement my mesh (tentatively calling it 
> "TREEMESH" but open to suggestions, I am so uncreative with names 
> >.<).  So far a couple (simple) questions have come up.
>
> The first question is, how much information is too much for a header 
> placed on a message?  Currently I am storing the hop record for each 
> message in the header.  Which contains the JGroups address of each 
> hop, and 3 shorts for each hop.  Would having a potentially largish 
> header be an issue (biggest natural concern of mine would be 
> fragmentation)?

Yes, that might become an issue: fragmentation (FRAG2) only uses the 
payload size (Message.getLength()) to determine whether to fragment or 
not. The getLength() method is very fast compared to size() which 
computes the size of he entire message when marshalled (including headers).

If you only store 3 addresses in the header, then that's fine, but if 
you store all addresses, then you'll run into an issue. Well, anyway, 
you're only storing 3 addresses plus change, so that's definitely not an 
issue !

For example, in STABLE, we send digests around, which contain all 
addresses of all members in the cluster. This has been identified [1] as 
an issue when we have large clusters (hundreds of nodes). For example, 
in our experiments on Infiniband (which has a 4K max message size), we 
had to add FRAG (which fragments the entire message) on top of the 
transport, but this is of course expensive.

One solution to mitigate this is to reduce the size of addresses, and a 
possible solution is the canonicalization of UUIDs to IDs (shorts), see 
[2] for details.

Having said that, you're not affected by this, as you only have a 
limited (and fixed !) number of addresses in the header.

> How should properties that should be defined as final be used?  
> Meaning, I have a couple properties that should be final, because they 
> are needed at init() time and after that should not be adjusted.  But 
> I would still like to allow them to be configurable on some level 
> before run time.  Should I define those as static finals, and expect 
> code changes for them to change (a user probably wont want to change 
> them anyways)?  Or is there a better way to set them so they can be 
> more flexible?  An example of these two values are:
>    * ProcessPeriod, period of wait time for regular run processes.  I 
> schedule a thread which regularly will look if it needs to establish 
> new connections, timeout dead peers, and remove duplicate connections 
> (which may likely occur with leaf connections due to two leaves both 
> deciding at the same time they want to connect to each other, this is 
> likely since they probably have the same internal models and thus both 
> decide they want a connection at the same time).  This can't be 
> adjusted because the value will be used at init time to schedule a 
> reoccurring task.
>    * sampleSize, this value determines how quickly we want to make 
> decisions vs how sure we want to be of our selves before any decision 
> or recommendations are made....with some changes this could be 
> adjusted at runtime, but I think that would be a mistake

Why would you want these to be final ? I'd only define constants as 
final, but properties should be able to be changed at run time. If you 
don't want this, define the properties as protected and make them 
read-only, e.g.

@Property(description="bla",writable=false)
protected long timeout;



> Initial discovery....I was hoping I could reuse one of the existing 
> discovery protocols like TCPPING.  But I am struggling to understand 
> how, and wondering if this is possible now.  Basically I was hoping 
> the initial join process could look something like this
> * Discovery finds a node that we can connect to
> * We connect to this node and now discovery goes hands off for further 
> work (meaning once we establish a connection to the first node, I 
> don't want this protocol layer to try and start connecting to more 
> nodes).
> * The TreeMESH protocol then exchanges information with this node, it 
> may remain connected, but most likely we will just be informed of a 
> different node we should connect to instead.
> * Once established in the mesh, if we are a leaf, the TreeMESH 
> protocol will identify other leafs that would be good choices and 
> connect to them too

Well, if you for example used TCPPING, you'd have a static list of nodes 
that TCPPING will try to contact. TCPGOSSIP could use external 
GossipRouter processes, and MPING/PING use IP multicasting. All you need 
to do here is to send down a FIND_INITIAL_MBRS / FIND_ALL_MBRS event, 
and discovery will return a list of found members to you.

What you then do is up to you, e.g. you could contact the first node in 
the list and ask it to join you etc.

If this doesn't fit your requirements, you could simply subclass 
Discovery and write a new discovery protocol. But I suppose one of the 
existing discovery protocols willl do.

> The problem is, I don't see how other protocols become aware of 
> discovery events (without the use of GMS which sends view updates, but 
> I don't think will work for my needs).  Am I correct in saying that 
> Event.CONNECT only represents this process joining the channel (not 
> establishing a connection to another node, or another node 
> establishing to us...if that is the case, how do we even know when 
> another node has connected to us?)?  As far as I can tell, I will need 
> to create my own discovery in order to accomplish what I described above?

Come again ? CONNECT tells the GMS protocol to join the cluster. The GMS 
protocol then sends a FIND_INITIAL_MBRS event down, to be handled by the 
discovery protocol. I assume you'd be the one to do this and process the 
response of the event, because you're the one who's joining. [3] 
describes the join process in a bit more detail.

> (p.s. while investigating this, I think I found a small logging bug.  
> I think line 499 (CVS head) in JChannel should be 
> this.cluster_name...i was thinking this might return the wrong name if 
> your trying to connect to a different channel than your already 
> connected to)

thx, fixed !

> I am not sure how to tell the transport layer to establish a 
> connection directly to another node.  I assume this would be an event, 
> but I can't tell what kind of event would do such a thing.

By simply sending a message to an Address, e.g.

Address random=(Address)Util.pickRandomElement(view.getMembers());

channel.send(random, null, "bla");

If TCP doesn't have a connection to 'random' yet, it'll create one. The 
event would be

Message msg=new Message(random, null, "bla");
down_prot.down(new Event(Event.MSG, msg));

> If you have time, I would appreciate a quick look at the start of my 
> main TREEMESH.java protocol (attached).  Right now it is just a 
> template with lots of TODO statements.  I am sure there is much more 
> definition that needs to be added in here still.  But I was hoping you 
> guys could take a quick look through it and let me know if I am on the 
> right track, or what recommendations you have.  My biggest concern 
> right now is how to integrate in the protocol stack properly.  I am 
> not sure what events I can/will get, or what events I should send.

Looks good, a few comments though:

    * As you mentioned, unless you want the properties to be changeable
      after starting (via JMX or probe.sh for example), you should mark
      then as "writable=false".
    * init() should also have a matching destroy(), which cancels the
      task. Not *strictly* needed, because TP.stop() will stop the
      timer, and this cancels all tasks, but it is considered good
      programming practice to do this
    * I would make MsgHop implement Streamable, so it uses less memory


> Feel free to ask any questions you might have about the directions I 
> have been taking this.  Thanks for spending some time to help me get 
> some momentum to this.


[1] https://jira.jboss.org/browse/JGRP-100, last comment
[2] https://jira.jboss.org/browse/JGRP-931
[3] http://community.jboss.org/wiki/HandleJoinProblem



-- 
Bela Ban
Lead JGroups / Clustering Team
JBoss

------------------------------------------------------------------------------
Sell apps to millions through the Intel(R) Atom(Tm) Developer Program
Be part of this innovative community and reach millions of netbook users 
worldwide. Take advantage of special opportunities to increase revenue and 
speed time-to-market. Join now, and jumpstart your future.
http://p.sf.net/sfu/intel-atom-d2d
_______________________________________________
Javagroups-development mailing list