Re: [jgroups-dev] Advice on best practices for implementing a new protocol

Things have been going great. I have a partially functional version
working now. Although I still have lots of questions (only one new one
that I have not emailed about before). Also lots more refining needs
to happen.

Bela, do you think you would have a time for me to talk to you on the
phone? I was hoping we could have another little meeting where I go
over what I have built, the structure, ask you lots of questions, and
get advice. When would be a good time for you? I don't mind waking up
early if it makes it more convenient.

Other people would also be welcome to this discussion/meeting if
interested.

If you have time to respond to some of my questions, it would be nice
as it would allow me to continue work until this meeting (and thus can
show a more complete solution then). Just to recap, the still
outstanding questions are (see previous emails for more details):

* Why are message headers included in every fragmented message, do we
still have an issue here since the hops/addresses stored in the message
are not bounded (as you thought they were)

* Comments on how I am using the up/down functions, what would good
return values be since one event may spawn multiple events?

* How can I close a connection?

But I highly suggest reading my previous two emails as they also
contain comments/suggestions/and much more detail.

Cheers

Mike Jensen wrote:

A couple more notes in addition to the email I sent earlier this
morning...(sorry to be throwing so much at you, hopefully your not
thinking this would just be easier to implement yourself, hehe)

Because of the way that connections are established (by sending
messages to a destination that not already connected to), I realized I
might need to handle this different. I would like your feedback on
this proposal:

The idea is that when the protocol gets an incoming message from the
application, we remove the dest and instead place the final destination
inside the mesh header. Then we send out event(s) for the nodes
connected to us that should receive this message, except we replace
destination to be those nodes directly. So when sketched out, it might
look something like this for up/down:

public Object down(Event event) {

switch (event.getType()) {

case Event.MSG:

// add our header on the message before passing on

Message msg = (Message)event.getArg();

msg.putHeaderIfAbsent(getId(), new TreeMeshHeader(local_addr,

(msg.getDest() == null || msg.getDest().isMulticastAddress() ? null :
msg.getDest()), // null here represents that this message should be
seen by the entire cluster

meshModel.getAvgHops(),

meshModel.peerConnectionCount(),

meshModel.leafConnectionCount()));

// call down_prot.down(event) with destination to be the
connected nodes this should be sent to next

// return null? not sure what to return since we are spawning
multiple down prot events from this single event

break;

case Event.DISCONNECT:

return down_prot.down(event);

case Event.SET_LOCAL_ADDRESS:

local_addr = (Address) event.getArg();

return down_prot.down(event);

default:

return down_prot.down(event);

}

}

public Object up(Event event) {

switch (event.getType()) {

case Event.MSG:

Message msg = (Message)event.getArg();

TreeMeshHeader hdr = (TreeMeshHeader)msg.getHeader(getId());

// analyze the message for more mesh structure information

meshModel.analyzeMessageHopRecord(hdr.hopRecord);

if (hdr.destination == null) {

hdr.addHop(local_addr, meshModel.getAvgHops(),
meshModel.peerConnectionCount(), meshModel.leafConnectionCount());

// forward the message on to all connected nodes

List<Address> nodes =
meshModel.getAllConnectedNodes();

Iterator<Address> it = nodes.iterator();

while (it.hasNext()) {

msg.setDest(it.next());

down_prot.down(new Event(Event.MSG, msg));

}

// message will be routed to the application via
up_prot.up, as this was broadcast everyone should see it

} else if (hdr.destination.equals(local_addr)) {

// message will be routed to the application via up_prot.up

} else {

// TODO - attempt to route the message to the given
destination

return null; // return null because we were just acting
as a relay node, and thus don't want the application to see this
message?

}

}

return up_prot.up(event);

}

Thoughts on implementing routing like this?? I am concerned about what
would be valid return values when I am spawning multiple events from
one (and thus why I am just using null, which I doubt will work, but
not sure what it should be).

Moving on to a different question, as far as I can see there is no
mechanism to close an open connection. Is this correct? If so, any
suggestions on the best way to include this functionality? I am not
sure if getting a handle on the AbstractConnectionMap (or rather what
has implemented it) from within my protocol is the best idea. If that
is what is needed, maybe I should implement my own TCP protocol that
extends the existing TCP. Suggestions?

Lastly, since multiple TCP connections may be established between two
nodes (as described earlier, since two nodes probably have the same
internal models, there is a high probability that they will decide to
connect to each other at the same time), I need a way to distinguish
between two different connections of the same node. I have so far been
implementing everything by referencing to different nodes via their
"Address" object. Of course that does not work for the above
situation, because their address objects should return false in
.equals, even when they are of the same node (it returns false because
the ports should be different, and if they did return true you might
have a connection leak in AbstractConnectionMap).

So any suggestions on how to tell that two addresses are actually
connected to the same node? I don't want to do an IP comparison
because I worry it could cause confusion in situations where the nodes
are possibly behind a NAT (but maybe this is outside the scope of
JGroups for now?)...I am also not confident in the ability for this to
function correctly when using the SHARED_LOOPBACK.

If I can't do it from the Address object, is there some other object
handle you might recommend to reference nodes?

As always, thank you very much for your help.

Bela Ban wrote:

Mike Jensen wrote:

I started work today to implement my mesh
(tentatively calling it "TREEMESH" but open to suggestions, I am so
uncreative with names >.<). So far a couple (simple) questions
have come up.

The first question is, how much information is too much for a header
placed on a message? Currently I am storing the hop record for each
message in the header. Which contains the JGroups address of each hop,
and 3 shorts for each hop. Would having a potentially largish header
be an issue (biggest natural concern of mine would be fragmentation)?

Yes, that might become an issue: fragmentation (FRAG2) only uses the
payload size (Message.getLength()) to determine whether to fragment or
not. The getLength() method is very fast compared to size() which
computes the size of he entire message when marshalled (including
headers).

If you only store 3 addresses in the header, then that's fine, but if
you store all addresses, then you'll run into an issue. Well, anyway,
you're only storing 3 addresses plus change, so that's definitely not
an issue !

For example, in STABLE, we send digests around, which contain all
addresses of all members in the cluster. This has been identified [1]
as an issue when we have large clusters (hundreds of nodes). For
example, in our experiments on Infiniband (which has a 4K max message
size), we had to add FRAG (which fragments the entire message) on top
of the transport, but this is of course expensive.

One solution to mitigate this is to reduce the size of addresses, and a
possible solution is the canonicalization of UUIDs to IDs (shorts), see
[2] for details.

Having said that, you're not affected by this, as you only have a
limited (and fixed !) number of addresses in the header.

How should properties that should be
defined
as final be used? Meaning, I have a couple properties that should be
final, because they are needed at init() time and after that should not
be adjusted. But I would still like to allow them to be configurable
on some level before run time. Should I define those as static finals,
and expect code changes for them to change (a user probably wont want
to change them anyways)? Or is there a better way to set them so they
can be more flexible? An example of these two values are:

* ProcessPeriod, period of wait time for regular run processes. I
schedule a thread which regularly will look if it needs to establish
new connections, timeout dead peers, and remove duplicate connections
(which may likely occur with leaf connections due to two leaves both
deciding at the same time they want to connect to each other, this is
likely since they probably have the same internal models and thus both
decide they want a connection at the same time). This can't be
adjusted because the value will be used at init time to schedule a
reoccurring task.

* sampleSize, this value determines how quickly we want to make
decisions vs how sure we want to be of our selves before any decision
or recommendations are made....with some changes this could be adjusted
at runtime, but I think that would be a mistake

Why would you want these to be final ? I'd only define constants as
final, but properties should be able to be changed at run time. If you
don't want this, define the properties as protected and make them
read-only, e.g.

@Property(description="bla",writable=false)

protected long timeout;

Initial discovery....I was hoping I could
reuse one of the existing discovery protocols like TCPPING. But I am
struggling to understand how, and wondering if this is possible now.
Basically I was hoping the initial join process could look something
like this

* Discovery finds a node that we can connect to

* We connect to this node and now discovery goes hands off for further
work (meaning once we establish a connection to the first node, I don't
want this protocol layer to try and start connecting to more nodes).

* The TreeMESH protocol then exchanges information with this node, it
may remain connected, but most likely we will just be informed of a
different node we should connect to instead.

* Once established in the mesh, if we are a leaf, the TreeMESH protocol
will identify other leafs that would be good choices and connect to
them too

Well, if you for example used TCPPING, you'd have a static list of
nodes that TCPPING will try to contact. TCPGOSSIP could use external
GossipRouter processes, and MPING/PING use IP multicasting. All you
need to do here is to send down a FIND_INITIAL_MBRS / FIND_ALL_MBRS
event, and discovery will return a list of found members to you.

What you then do is up to you, e.g. you could contact the first node in
the list and ask it to join you etc.

If this doesn't fit your requirements, you could simply subclass
Discovery and write a new discovery protocol. But I suppose one of the
existing discovery protocols willl do.

The problem is, I don't see how other
protocols become aware of discovery events (without the use of GMS
which sends view updates, but I don't think will work for my needs).
Am I correct in saying that Event.CONNECT only represents this process
joining the channel (not establishing a connection to another node, or
another node establishing to us...if that is the case, how do we even
know when another node has connected to us?)? As far as I can tell, I
will need to create my own discovery in order to accomplish what I
described above?

Come again ? CONNECT tells the GMS protocol to join the cluster. The
GMS protocol then sends a FIND_INITIAL_MBRS event down, to be handled
by the discovery protocol. I assume you'd be the one to do this and
process the response of the event, because you're the one who's
joining. [3] describes the join process in a bit more detail.

(p.s. while investigating this, I think I
found a small logging bug. I think line 499 (CVS head) in JChannel
should be this.cluster_name...i was thinking this might return the
wrong name if your trying to connect to a different channel than your
already connected to)

thx, fixed !

I am not sure how to tell the transport
layer
to establish a connection directly to another node. I assume this
would be an event, but I can't tell what kind of event would do such a
thing.

By simply sending a message to an Address, e.g.

Address random=(Address)Util.pickRandomElement(view.getMembers());

channel.send(random, null, "bla");

If TCP doesn't have a connection to 'random' yet, it'll create one. The
event would be

Message msg=new Message(random, null, "bla");

down_prot.down(new Event(Event.MSG, msg));

If you have time, I would appreciate a
quick
look at the start of my main TREEMESH.java protocol (attached). Right
now it is just a template with lots of TODO statements. I am sure
there is much more definition that needs to be added in here still.
But I was hoping you guys could take a quick look through it and let me
know if I am on the right track, or what recommendations you have. My
biggest concern right now is how to integrate in the protocol stack
properly. I am not sure what events I can/will get, or what events I
should send.

Looks good, a few comments though:

* As you mentioned, unless you want the properties to be changeable

after starting (via JMX or probe.sh for example), you should mark

then as "writable=false".

* init() should also have a matching destroy(), which cancels the

task. Not *strictly* needed, because TP.stop() will stop the

timer, and this cancels all tasks, but it is considered good

programming practice to do this

* I would make MsgHop implement Streamable, so it uses less memory

Feel free to ask any questions you might
have
about the directions I have been taking this. Thanks for spending some
time to help me get some momentum to this.

[1] https://jira.jboss.org/browse/JGRP-100 ,
last comment

[2] https://jira.jboss.org/browse/JGRP-931

[3] http://community.jboss.org/wiki/HandleJoinProblem

------------------------------------------------------------------------------
Sell apps to millions through the Intel(R) Atom(Tm) Developer Program
Be part of this innovative community and reach millions of netbook users 
worldwide. Take advantage of special opportunities to increase revenue and 
speed time-to-market. Join now, and jumpstart your future.
http://p.sf.net/sfu/intel-atom-d2d

_______________________________________________
Javagroups-development mailing list