Re: [jgroups-dev] Advice on best practices for implementing a new protocol

Mike Jensen wrote:
>> If you only store 3 addresses in the header, then that's fine, but if
>> you store all addresses, then you'll run into an issue. Well, anyway,
>> you're only storing 3 addresses plus change, so that's definitely not
>> an issue !
> It very well could be more than 3 addresses. It will be 1 address + 3
> shorts for every hop. So if it is more than 3 hops, than it would be
> more than 3 addresses.

As long as you include 1 address for *all* nodes in your treemesh, 
that's still fine.

> But this begs the question. Why are headers
> included on every fragmented message? It seems like FRAG could just
> include it's own header (as it does), and then the first part could
> include the real headers for when the message is re-assembled. I would
> assume FRAG does not pass the message events on to other protocols until
> the message has been reassembled.

Good point; this is unnecessary. I created [1] to fix this in 2.11.



>> One solution to mitigate this is to reduce the size of addresses, and
>> a possible solution is the canonicalization of UUIDs to IDs (shorts),
>> see [2] for details.
>>
>> Having said that, you're not affected by this, as you only have a
>> limited (and fixed !) number of addresses in the header.
>>
> I think I am. I am unsure why you thought it would be only 3 addresses
> (because of only 3 connections each node maintains?). Although hops
> should remain low, they are unbounded as we scale, so I could see this
> being a problem for thousands of nodes.

OK, but with your flooding algorithm, it would still almost never be N, 
correct ? I assume it would be closer to log(N)...

Now, even if it was N, you could implement a simple hop count header, 
which includes only a view-id and a bitmap/bitset with the bits 
representing the ranks of the members (e.g. for {A,B,C}, C's rank is 2), 
and a bit that is set representing a member which has already seen the 
message.

This is an optimization and would require some sort of agreement 
/consensus protocol establishing a view-id, which is a list of members 
shared by all nodes.


>> Why would you want these to be final ? I'd only define constants as
>> final, but properties should be able to be changed at run time. If you
>> don't want this, define the properties as protected and make them
>> read-only, e.g.
>>
>> @Property(description="bla",writable=false)
>> protected long timeout;
>>
>>
> Does writable=false mean that it can be configured in a .xml file, but 
> not at runtime? 

Yes

> That's what I want. They should be able to be
> configured before the application has started up (meaning connected and
> init run, the process needs to be running to change the coded values).
> My concern is that if I make it so they can be adjusted at runtime,
> people will change them expecting a change in behavior, but since the
> values are only used at init time, that wont be the case. But I would
> like to maintain the flexibility for them to be changed before init has
> been run (which is why I agree that final is not the right solution).
> If that is not possible, then I will just set them as final constants.

If you decide one day to make these properties editable at runtime, you 
can simply remove the writable attribute from @Property...


>>> I am not sure how to tell the transport layer to establish a
>>> connection directly to another node. I assume this would be an
>>> event, but I can't tell what kind of event would do such a thing.
>> By simply sending a message to an Address, e.g.
>>
>> Address random=(Address)Util.pickRandomElement(view.getMembers());
>>
>> channel.send(random, null, "bla");
>>
>> If TCP doesn't have a connection to 'random' yet, it'll create one.
>> The event would be
>>
>> Message msg=new Message(random, null, "bla");
>> down_prot.down(new Event(Event.MSG, msg));
> Brilliant, that will work perfectly! But what is the best way to be
> aware that a new node has connected to us? 

The node needs to send a message, similar to the regular JOIN-REQ in GMS.

> The nodes already have to  exchange information between each other 
> before we allow them into the tree.

Looks similar to what Discovery (subclass) protocols do: the physical IP 
address, logical address and logical name is exchanged here.

> Specifically the most important information they provide us is if
> they are connecting to us with the intent to have this be a peer
> connection (a connection that is part of the tree structure formation),
> or a leaf connection (more temporary and based on distance instead of
> tree structure). Once we know their intention, we ether tell them to go
> away, and suggest a different node. Or we accept them, and respond with
> similar information that they sent us originally.
>
> So what that brings me to is, that connecting to them is easy. But
> nodes are in a limbo state after they connect, we don't immediately
> consider them part of the group until after this handshake. So how can
> I detect a node has connected, but is still left in this limbo state?

I'd not dissemniate information about them in MESH messages, until their 
not in limbo state anymore.

> The one possible solution I can think of:
> * Listen to the content of all messages being passed, and be looking
> specifically for these types of handshake messages. If we get one, then
> we perform the handshake, and we are happy.

Why? A JOIN-REQ would just be a different message from the other (e.g. 
MESH routing or regular) messages.

> Problems:
> * Load, I hate to analyze every message content that may or may not be
> relevant....it seems like a waste when

A JOIN-REQ is only sent to a selected peer, and not to everyone. That 
peer might respond with a REDIRECT-TO(P), but then the joiner simply 
sends a new JOIN-REQ to P.

>
>>> Feel free to ask any questions you might have about the directions I
>>> have been taking this. Thanks for spending some time to help me get
>>> some momentum to this.
>>
>> [1] https://jira.jboss.org/browse/JGRP-100, last comment
>> [2] https://jira.jboss.org/browse/JGRP-931
>> [3] http://community.jboss.org/wiki/HandleJoinProblem
>>
> [2] - I am not sure this is the best idea...I have thought about
> something like this, as it would greatly improve merging two partitions
> in my design. But having anything that must be agreed on by the whole
> cluster has too much performance implications. The idea mentioned of
> the coordinator providing them may be better.

To have a coordinator also implies having to run an agreement / 
consensus protocol...

> How this could work in my protocol would be something like this:
> * The node connects to a random node (or preferably the coordinator, 
> but both cases should be possible)
> * The node contacts the coordinator, asking for the next ID before it
> gives it to the new node, and tells him where to connect and join the
> tree structure

If you leave off the direct connections between leaf nodes, you have a 
tree. With a tree, you could grab the ID directly from your parent node, 
which has authority to dispense IDs...

E.g the root node is always 0, the children start with 10,11,12 etc, 
their children start with 100,101,102 etc. This scheme requires your 
tree to not to have too many levels though... Plus, the ID of a crashed 
peer cannot be reused. And tree reorgs are more difficult...

In the worst case, create an external ID service, which returns 
monotonically increasing IDs.. Not very p2p like though...


> Possible issues -
> * A failed coordinator then means that join time is much longer.
> * What if a partition has occurred, and then nodes join? 

Yep, this is the biggest scenario I have to handle too in JGRP-931. If 
we have peers with identical IDs in 2 partitions, and then the 
partitions heal. My current thinking is that peers will always be able 
to handle both UUIDs and IDs (e.g. do comparisons between them), and if 
a partition occurs and heals, we have to (a) fall back to UUIDs, (b) 
identify duplicate IDs and (c) re-allocate those. I haven't spent much 
time on this, so these ideas are still somewhat immature...


> Meaning that
> the group has been split in half, and new nodes have joined on both
> halves. When they are merged back together, how do we resolve or ensure
> there are no conflicting ID's? From experience, changing an ID is a
> tricky proposition, and maybe not realistic....but if a solution to this
> could be found, then there are other possible solutions I can think of 
> too.
>
> Thoughts?
>
> Thanks for all your input, I will continue working on this today. I
> will let you know if I have more questions.

[1] https://jira.jboss.org/browse/JGRP-1230

-- 
Bela Ban
Lead JGroups / Clustering Team
JBoss

------------------------------------------------------------------------------
Sell apps to millions through the Intel(R) Atom(Tm) Developer Program
Be part of this innovative community and reach millions of netbook users 
worldwide. Take advantage of special opportunities to increase revenue and 
speed time-to-market. Join now, and jumpstart your future.
http://p.sf.net/sfu/intel-atom-d2d
_______________________________________________
Javagroups-development mailing list