Re: [jgroups-dev] Advice on best practices for implementing a new protocol

Responses inline....

Bela Ban wrote:
>
>
> Mike Jensen wrote:
>> I started work today to implement my mesh (tentatively calling it 
>> "TREEMESH" but open to suggestions, I am so uncreative with names 
>> >.<).  So far a couple (simple) questions have come up.
>>
>> The first question is, how much information is too much for a header 
>> placed on a message?  Currently I am storing the hop record for each 
>> message in the header.  Which contains the JGroups address of each 
>> hop, and 3 shorts for each hop.  Would having a potentially largish 
>> header be an issue (biggest natural concern of mine would be 
>> fragmentation)?
>
> Yes, that might become an issue: fragmentation (FRAG2) only uses the 
> payload size (Message.getLength()) to determine whether to fragment or 
> not. The getLength() method is very fast compared to size() which 
> computes the size of he entire message when marshalled (including 
> headers).
>
> If you only store 3 addresses in the header, then that's fine, but if 
> you store all addresses, then you'll run into an issue. Well, anyway, 
> you're only storing 3 addresses plus change, so that's definitely not 
> an issue !
It very well could be more than 3 addresses.  It will be 1 address + 3 
shorts for every hop.  So if it is more than 3 hops, than it would be 
more than 3 addresses.  But this begs the question.  Why are headers 
included on every fragmented message?  It seems like FRAG could just 
include it's own header (as it does), and then the first part could 
include the real headers for when the message is re-assembled.  I would 
assume FRAG does not pass the message events on to other protocols until 
the message has been reassembled.
>
> For example, in STABLE, we send digests around, which contain all 
> addresses of all members in the cluster. This has been identified [1] 
> as an issue when we have large clusters (hundreds of nodes). For 
> example, in our experiments on Infiniband (which has a 4K max message 
> size), we had to add FRAG (which fragments the entire message) on top 
> of the transport, but this is of course expensive.
>
> One solution to mitigate this is to reduce the size of addresses, and 
> a possible solution is the canonicalization of UUIDs to IDs (shorts), 
> see [2] for details.
>
> Having said that, you're not affected by this, as you only have a 
> limited (and fixed !) number of addresses in the header.
>
I think I am.  I am unsure why you thought it would be only 3 addresses 
(because of only 3 connections each node maintains?).  Although hops 
should remain low, they are unbounded as we scale, so I could see this 
being a problem for thousands of nodes.  But maybe this is just an issue 
we should try to tackle later?
>> How should properties that should be defined as final be used?  
>> Meaning, I have a couple properties that should be final, because 
>> they are needed at init() time and after that should not be 
>> adjusted.  But I would still like to allow them to be configurable on 
>> some level before run time.  Should I define those as static finals, 
>> and expect code changes for them to change (a user probably wont want 
>> to change them anyways)?  Or is there a better way to set them so 
>> they can be more flexible?  An example of these two values are:
>>    * ProcessPeriod, period of wait time for regular run processes.  I 
>> schedule a thread which regularly will look if it needs to establish 
>> new connections, timeout dead peers, and remove duplicate connections 
>> (which may likely occur with leaf connections due to two leaves both 
>> deciding at the same time they want to connect to each other, this is 
>> likely since they probably have the same internal models and thus 
>> both decide they want a connection at the same time).  This can't be 
>> adjusted because the value will be used at init time to schedule a 
>> reoccurring task.
>>    * sampleSize, this value determines how quickly we want to make 
>> decisions vs how sure we want to be of our selves before any decision 
>> or recommendations are made....with some changes this could be 
>> adjusted at runtime, but I think that would be a mistake
>
> Why would you want these to be final ? I'd only define constants as 
> final, but properties should be able to be changed at run time. If you 
> don't want this, define the properties as protected and make them 
> read-only, e.g.
>
> @Property(description="bla",writable=false)
> protected long timeout;
>
>
Does writable=false mean that it can be configured in a .xml file, but 
not at runtime?  That's what I want.  They should be able to be 
configured before the application has started up (meaning connected and 
init run, the process needs to be running to change the coded values).  
My concern is that if I make it so they can be adjusted at runtime, 
people will change them expecting a change in behavior, but since the 
values are only used at init time, that wont be the case.  But I would 
like to maintain the flexibility for them to be changed before init has 
been run (which is why I agree that final is not the right solution).  
If that is not possible, then I will just set them as final constants.
>
>> Initial discovery....I was hoping I could reuse one of the existing 
>> discovery protocols like TCPPING.  But I am struggling to understand 
>> how, and wondering if this is possible now.  Basically I was hoping 
>> the initial join process could look something like this
>> * Discovery finds a node that we can connect to
>> * We connect to this node and now discovery goes hands off for 
>> further work (meaning once we establish a connection to the first 
>> node, I don't want this protocol layer to try and start connecting to 
>> more nodes).
>> * The TreeMESH protocol then exchanges information with this node, it 
>> may remain connected, but most likely we will just be informed of a 
>> different node we should connect to instead.
>> * Once established in the mesh, if we are a leaf, the TreeMESH 
>> protocol will identify other leafs that would be good choices and 
>> connect to them too
>
> Well, if you for example used TCPPING, you'd have a static list of 
> nodes that TCPPING will try to contact. TCPGOSSIP could use external 
> GossipRouter processes, and MPING/PING use IP multicasting. All you 
> need to do here is to send down a FIND_INITIAL_MBRS / FIND_ALL_MBRS 
> event, and discovery will return a list of found members to you.
>
> What you then do is up to you, e.g. you could contact the first node 
> in the list and ask it to join you etc.
>
> If this doesn't fit your requirements, you could simply subclass 
> Discovery and write a new discovery protocol. But I suppose one of the 
> existing discovery protocols willl do.
>
Ah, okay, I will look into this more with that information now.
>> The problem is, I don't see how other protocols become aware of 
>> discovery events (without the use of GMS which sends view updates, 
>> but I don't think will work for my needs).  Am I correct in saying 
>> that Event.CONNECT only represents this process joining the channel 
>> (not establishing a connection to another node, or another node 
>> establishing to us...if that is the case, how do we even know when 
>> another node has connected to us?)?  As far as I can tell, I will 
>> need to create my own discovery in order to accomplish what I 
>> described above?
>
> Come again ? CONNECT tells the GMS protocol to join the cluster. The 
> GMS protocol then sends a FIND_INITIAL_MBRS event down, to be handled 
> by the discovery protocol. I assume you'd be the one to do this and 
> process the response of the event, because you're the one who's 
> joining. [3] describes the join process in a bit more detail.
>> (p.s. while investigating this, I think I found a small logging bug.  
>> I think line 499 (CVS head) in JChannel should be 
>> this.cluster_name...i was thinking this might return the wrong name 
>> if your trying to connect to a different channel than your already 
>> connected to)
>
> thx, fixed !
>
>> I am not sure how to tell the transport layer to establish a 
>> connection directly to another node.  I assume this would be an 
>> event, but I can't tell what kind of event would do such a thing.
>
> By simply sending a message to an Address, e.g.
>
> Address random=(Address)Util.pickRandomElement(view.getMembers());
>
> channel.send(random, null, "bla");
>
> If TCP doesn't have a connection to 'random' yet, it'll create one. 
> The event would be
>
> Message msg=new Message(random, null, "bla");
> down_prot.down(new Event(Event.MSG, msg));
Brilliant, that will work perfectly!  But what is the best way to be 
aware that a new node has connected to us?  The nodes already have to 
exchange information between each other before we allow them into the 
tree.  Specifically the most important information they provide us is if 
they are connecting to us with the intent to have this be a peer 
connection (a connection that is part of the tree structure formation), 
or a leaf connection (more temporary and based on distance instead of 
tree structure).  Once we know their intention, we ether tell them to go 
away, and suggest a different node.  Or we accept them, and respond with 
similar information that they sent us originally.

So what that brings me to is, that connecting to them is easy.  But 
nodes are in a limbo state after they connect, we don't immediately 
consider them part of the group until after this handshake.  So how can 
I detect a node has connected, but is still left in this limbo state?

The one possible solution I can think of:
* Listen to the content of all messages being passed, and be looking 
specifically for these types of handshake messages.  If we get one, then 
we perform the handshake, and we are happy.

Problems:
* Load, I hate to analyze every message content that may or may not be 
relevant....it seems like a waste when
* If a node connects, but starts sending messages that don't contain 
this handshake (or if the handshake is lost due to unreliable transport) 
the following will occur:
    * The node wont be part of the group, and thus probably wont receive 
messages back or know about the other members
    * The connection will never be closed, because we are unaware of 
this particular nodes state
    * Possible weird things because it is working outside expectations

But if this is the only realistic option, then I will consider it when I 
code, and find ways to control this risk.
>
>> If you have time, I would appreciate a quick look at the start of my 
>> main TREEMESH.java protocol (attached).  Right now it is just a 
>> template with lots of TODO statements.  I am sure there is much more 
>> definition that needs to be added in here still.  But I was hoping 
>> you guys could take a quick look through it and let me know if I am 
>> on the right track, or what recommendations you have.  My biggest 
>> concern right now is how to integrate in the protocol stack 
>> properly.  I am not sure what events I can/will get, or what events I 
>> should send.
>
> Looks good, a few comments though:
>
>    * As you mentioned, unless you want the properties to be changeable
>      after starting (via JMX or probe.sh for example), you should mark
>      then as "writable=false".
>    * init() should also have a matching destroy(), which cancels the
>      task. Not *strictly* needed, because TP.stop() will stop the
>      timer, and this cancels all tasks, but it is considered good
>      programming practice to do this
>    * I would make MsgHop implement Streamable, so it uses less memory
>
Good recommendations, thanks!
>
>> Feel free to ask any questions you might have about the directions I 
>> have been taking this.  Thanks for spending some time to help me get 
>> some momentum to this.
>
>
> [1] https://jira.jboss.org/browse/JGRP-100, last comment
> [2] https://jira.jboss.org/browse/JGRP-931
> [3] http://community.jboss.org/wiki/HandleJoinProblem
>
[2] - I am not sure this is the best idea...I have thought about 
something like this, as it would greatly improve merging two partitions 
in my design.  But having anything that must be agreed on by the whole 
cluster has too much performance implications.  The idea mentioned of 
the coordinator providing them may be better.  How this could work in my 
protocol would be something like this:
* The node connects to a random node (or preferably the coordinator, but 
both cases should be possible)
* The node contacts the coordinator, asking for the next ID before it 
gives it to the new node, and tells him where to connect and join the 
tree structure

Possible issues -
* A failed coordinator then means that join time is much longer.
* What if a partition has occurred, and then nodes join?  Meaning that 
the group has been split in half, and new nodes have joined on both 
halves.  When they are merged back together, how do we resolve or ensure 
there are no conflicting ID's?  From experience, changing an ID is a 
tricky proposition, and maybe not realistic....but if a solution to this 
could be found, then there are other possible solutions I can think of too.

Thoughts?

Thanks for all your input, I will continue working on this today.  I 
will let you know if I have more questions.

------------------------------------------------------------------------------
Sell apps to millions through the Intel(R) Atom(Tm) Developer Program
Be part of this innovative community and reach millions of netbook users 
worldwide. Take advantage of special opportunities to increase revenue and 
speed time-to-market. Join now, and jumpstart your future.
http://p.sf.net/sfu/intel-atom-d2d
_______________________________________________
Javagroups-development mailing list