Re: [jgroups-dev] Advice on best practices for implementing a new protocol

Thanks for all your feedback.

A few responses inline, but we still have more to discuss, mostly what i 
had described in my other email I sent (if it has gotten confusing or 
lost, let me know and I can redescribe the remaining problems, the 
biggest one is how do I disconnect nodes, but other questions also 
remain).  But I would like to find a time we can talk on the phone too.  
Would Monday be a realistic option for that for you?  Maybe Tuesday if not?

Bela Ban wrote:
>
>
> Mike Jensen wrote:
>>> If you only store 3 addresses in the header, then that's fine, but if
>>> you store all addresses, then you'll run into an issue. Well, anyway,
>>> you're only storing 3 addresses plus change, so that's definitely not
>>> an issue !
>> It very well could be more than 3 addresses. It will be 1 address + 3
>> shorts for every hop. So if it is more than 3 hops, than it would be
>> more than 3 addresses.
>
> As long as you include 1 address for *all* nodes in your treemesh, 
> that's still fine.
>
>> But this begs the question. Why are headers
>> included on every fragmented message? It seems like FRAG could just
>> include it's own header (as it does), and then the first part could
>> include the real headers for when the message is re-assembled. I would
>> assume FRAG does not pass the message events on to other protocols until
>> the message has been reassembled.
>
> Good point; this is unnecessary. I created [1] to fix this in 2.11.
>
>
>
>>> One solution to mitigate this is to reduce the size of addresses, and
>>> a possible solution is the canonicalization of UUIDs to IDs (shorts),
>>> see [2] for details.
>>>
>>> Having said that, you're not affected by this, as you only have a
>>> limited (and fixed !) number of addresses in the header.
>>>
>> I think I am. I am unsure why you thought it would be only 3 addresses
>> (because of only 3 connections each node maintains?). Although hops
>> should remain low, they are unbounded as we scale, so I could see this
>> being a problem for thousands of nodes.
>
> OK, but with your flooding algorithm, it would still almost never be 
> N, correct ? I assume it would be closer to log(N)...
>
> Now, even if it was N, you could implement a simple hop count header, 
> which includes only a view-id and a bitmap/bitset with the bits 
> representing the ranks of the members (e.g. for {A,B,C}, C's rank is 
> 2), and a bit that is set representing a member which has already seen 
> the message.
>
> This is an optimization and would require some sort of agreement 
> /consensus protocol establishing a view-id, which is a list of members 
> shared by all nodes.
>
>
You are correct, but nothing currently ensures this.  I can't imagine 
how it would happen, but I wont say it is impossible unless I 
specifically design it not to happen.
>>> Why would you want these to be final ? I'd only define constants as
>>> final, but properties should be able to be changed at run time. If you
>>> don't want this, define the properties as protected and make them
>>> read-only, e.g.
>>>
>>> @Property(description="bla",writable=false)
>>> protected long timeout;
>>>
>>>
>> Does writable=false mean that it can be configured in a .xml file, 
>> but not at runtime? 
>
> Yes
>
>> That's what I want. They should be able to be
>> configured before the application has started up (meaning connected and
>> init run, the process needs to be running to change the coded values).
>> My concern is that if I make it so they can be adjusted at runtime,
>> people will change them expecting a change in behavior, but since the
>> values are only used at init time, that wont be the case. But I would
>> like to maintain the flexibility for them to be changed before init has
>> been run (which is why I agree that final is not the right solution).
>> If that is not possible, then I will just set them as final constants.
>
> If you decide one day to make these properties editable at runtime, 
> you can simply remove the writable attribute from @Property...
>
>
>>>> I am not sure how to tell the transport layer to establish a
>>>> connection directly to another node. I assume this would be an
>>>> event, but I can't tell what kind of event would do such a thing.
>>> By simply sending a message to an Address, e.g.
>>>
>>> Address random=(Address)Util.pickRandomElement(view.getMembers());
>>>
>>> channel.send(random, null, "bla");
>>>
>>> If TCP doesn't have a connection to 'random' yet, it'll create one.
>>> The event would be
>>>
>>> Message msg=new Message(random, null, "bla");
>>> down_prot.down(new Event(Event.MSG, msg));
>> Brilliant, that will work perfectly! But what is the best way to be
>> aware that a new node has connected to us? 
>
> The node needs to send a message, similar to the regular JOIN-REQ in GMS.
>
I have another solution to this that does not require analyzing the 
message.  In each header I have an enum which states which state the 
message is in:
initialHandshake, responseHandshake (sent by the receiver of the 
initialHandshake), heartbeat, and applicationMsg

I would like to show you the design of my code, and maybe we can discuss 
if there is a better way to do this.  But I think my solution might be 
just fine.
>> The nodes already have to  exchange information between each other 
>> before we allow them into the tree.
>
> Looks similar to what Discovery (subclass) protocols do: the physical 
> IP address, logical address and logical name is exchanged here.
>
>> Specifically the most important information they provide us is if
>> they are connecting to us with the intent to have this be a peer
>> connection (a connection that is part of the tree structure formation),
>> or a leaf connection (more temporary and based on distance instead of
>> tree structure). Once we know their intention, we ether tell them to go
>> away, and suggest a different node. Or we accept them, and respond with
>> similar information that they sent us originally.
>>
>> So what that brings me to is, that connecting to them is easy. But
>> nodes are in a limbo state after they connect, we don't immediately
>> consider them part of the group until after this handshake. So how can
>> I detect a node has connected, but is still left in this limbo state?
>
> I'd not dissemniate information about them in MESH messages, until 
> their not in limbo state anymore.
>
I think I have a solution to this that I would like to show you
>> The one possible solution I can think of:
>> * Listen to the content of all messages being passed, and be looking
>> specifically for these types of handshake messages. If we get one, then
>> we perform the handshake, and we are happy.
>
> Why? A JOIN-REQ would just be a different message from the other (e.g. 
> MESH routing or regular) messages.
>
Your right, this can be avoided
>> Problems:
>> * Load, I hate to analyze every message content that may or may not be
>> relevant....it seems like a waste when
>
> A JOIN-REQ is only sent to a selected peer, and not to everyone. That 
> peer might respond with a REDIRECT-TO(P), but then the joiner simply 
> sends a new JOIN-REQ to P.
>
>>
>>>> Feel free to ask any questions you might have about the directions I
>>>> have been taking this. Thanks for spending some time to help me get
>>>> some momentum to this.
>>>
>>> [1] https://jira.jboss.org/browse/JGRP-100, last comment
>>> [2] https://jira.jboss.org/browse/JGRP-931
>>> [3] http://community.jboss.org/wiki/HandleJoinProblem
>>>
>> [2] - I am not sure this is the best idea...I have thought about
>> something like this, as it would greatly improve merging two partitions
>> in my design. But having anything that must be agreed on by the whole
>> cluster has too much performance implications. The idea mentioned of
>> the coordinator providing them may be better.
>
> To have a coordinator also implies having to run an agreement / 
> consensus protocol...
>
good point, I will need to think more about this
>> How this could work in my protocol would be something like this:
>> * The node connects to a random node (or preferably the coordinator, 
>> but both cases should be possible)
>> * The node contacts the coordinator, asking for the next ID before it
>> gives it to the new node, and tells him where to connect and join the
>> tree structure
>
> If you leave off the direct connections between leaf nodes, you have a 
> tree. With a tree, you could grab the ID directly from your parent 
> node, which has authority to dispense IDs...
>
> E.g the root node is always 0, the children start with 10,11,12 etc, 
> their children start with 100,101,102 etc. This scheme requires your 
> tree to not to have too many levels though... Plus, the ID of a 
> crashed peer cannot be reused. And tree reorgs are more difficult...
>
> In the worst case, create an external ID service, which returns 
> monotonically increasing IDs.. Not very p2p like though...
>
>
>> Possible issues -
>> * A failed coordinator then means that join time is much longer.
>> * What if a partition has occurred, and then nodes join? 
>
> Yep, this is the biggest scenario I have to handle too in JGRP-931. If 
> we have peers with identical IDs in 2 partitions, and then the 
> partitions heal. My current thinking is that peers will always be able 
> to handle both UUIDs and IDs (e.g. do comparisons between them), and 
> if a partition occurs and heals, we have to (a) fall back to UUIDs, 
> (b) identify duplicate IDs and (c) re-allocate those. I haven't spent 
> much time on this, so these ideas are still somewhat immature...
>
>
>> Meaning that
>> the group has been split in half, and new nodes have joined on both
>> halves. When they are merged back together, how do we resolve or ensure
>> there are no conflicting ID's? From experience, changing an ID is a
>> tricky proposition, and maybe not realistic....but if a solution to this
>> could be found, then there are other possible solutions I can think 
>> of too.
>>
>> Thoughts?
>>
>> Thanks for all your input, I will continue working on this today. I
>> will let you know if I have more questions.
>
> [1] https://jira.jboss.org/browse/JGRP-1230
>


------------------------------------------------------------------------------
Sell apps to millions through the Intel(R) Atom(Tm) Developer Program
Be part of this innovative community and reach millions of netbook users 
worldwide. Take advantage of special opportunities to increase revenue and 
speed time-to-market. Join now, and jumpstart your future.
http://p.sf.net/sfu/intel-atom-d2d
_______________________________________________
Javagroups-development mailing list