Re: [jgroups-dev] TreeMesh progress and questions

Hi Mike,

I applied your changes and tried out treeMesh_NIO.xml with Draw (2 on my 
Linux box and 2 on my Mac). The draw instances were able to find each 
other, but the views shown never stabilized; meaning I never got to 4 
members. Also, sometimes a node would not be able to communicate with 
the other nodes, for a few secs until the situation got cleared up...


Comments inline


Mike Jensen wrote:
> I know it has been a while since you have heard from me. But I been
> continuing to work on everything. At the bottom I have a link to
> download the code for you to examine.
>
> I am in the process of creating a chat application that can run on top
> of this. I have the basics of the chat application done, and it seems
> to be working just fine on top of the TreeMesh. I still have other
> things I want to do for it though:
> * Implement meshGUI in the chat application, so that you can actually
> see the connection structures that are being used for your chat session

Good !

> * Implement an IRC bridge, just to show an example of how a highly
> strong coherency application can work together with something like what
> I have built

To interact with real IRC ! That'd be cool !

> First a few notes about the implementation (so much I could write here,
> I just outlined some bigger items):
>
> * It does look like I have to keep track of what messages we have seen.
> Because messages are routed to us via multiple paths, I need to keep a
> list of longs which represent random msgID's and ignore those we have
> already seen. 

Why random IDs ? Why not simply have the sender use monotonically 
increasing IDs ?

You could maintain the highest seqno received from each node, and 
discard any message with a seqno which is lower than it. Whenever you 
receive a higher seqno, update that value in the hashmap.

This of course will lead to message loss in certain cases, e.g. if you 
receive messages 5 8 6 5 7 9 from P, you'll only deliver to the 
application messages 5, 8 and 9 (and discard 6, 5 and 7). However, this 
should not be such a big issue as you're mainly using TCP as transport, 
which guarantees ordering anyway.

So keeping the highest seqnos seems to be a very simple and efficient 
way of weeding out duplicates.


> I don't particularly like this because:
> * We now have a list that can get large (but is bounded)

Not with the above mechanism

> * We need to transmit another long with each message that could be a 
> broadcast msg

Yes. No big deal though, is it ?

> * Because we are transmitting a long vs a UUID, I am worried that we
> could end up dropping msgs if the history is allowed to grow too large
> (but is a configurable value)....is this worth the extra transmission
> overhead?
> Do you guys have any advice on possibly better ways to handle this?

See above. However, this would not work with UDP datagrams, and there 
you *might* actually incur message loss (versus using TCP).

> * I have created a new MeshView which extends View. So far this seems
> to work just fine as long as the application is aware that it could be
> getting a MeshView instead of a View (assuming the application wants to
> take advantage of the mesh structures).

Hmm. Another way of doing this is to use the payload in View. Well, 
actually I've deprecated use of the payload, but if there is a real 
need, we can reinstate it... This would obliviate the need for MeshView. 
Well, actually, maybe I like MeshView more after all, so I can go 
forward with deprecating payload !

> Although I can't seem to get the unicast protocol to work with my tree 
> protocol, and I suspect the
> issue may be because of this. But I have spent almost no time to 
> diagnose the problem.
UNICAST / UNICAST2 should work without views anyway ! Although it would 
be good to pass a view to them every now and then, so they can clean out 
connections to / from old members. Note that UNICAST2 doesn't use any acks.


> * I separated failure detection and merging based on the suggestions
> from Bela. Although this is not heavily tested, I think the merge
> algorithm that Bela suggested will work great, and seems very simple.

OK

> * Right now I have a few situations where nodes are timing out when they
> should not be. I have not spent any time to diagnose why those
> conditions might occur.

OK, maybe that's what I've observed when I ran your code.

> And now for the question!
>
> My biggest issue is that I have extended UUID with a new class
> "MeshIdentifier" that I want to function as the Address. The problem is
> that despite this trying to mimic an Address as close as I can figure
> out, I can't seem to get messages to actually route with this address.
> Further more, I need to figure out how to get this Address to be
> integrated deeper into JGroups so that when you call
> JChannel.getLocalAddress() it returns this UUID form. The reason this
> is important is because I want Addresses to be unique per instance, so
> that if machines span multiple datacenters than we are able to uniquely
> identify them even if they have the same IP/port. Another example of
> such a situation would be running multiple instances of the process on
> the same machine. Suggestions on how to better handle this? I invite
> you to look at my implementation, but right now MeshIdentifier just
> stores the real address + the super class UUID, and i just call
> .getAddress() before I route any messages. This is less than ideal as
> it means that I may or may not be sending the message to the node I
> think I am.

Not sure I understand what you're trying to achieve here. A UUID maps to 
a PhysicalAddress (IP address) and is always unique over space and time, 
also across different data centers. When we send a message to a UUID, we 
look up the physical address associated with that UUID and grab the 
connection to that physical address (destination).

The mapping between UUIDs and physical addresses is returned by the 
discovery protocol (TCPPING in your example).

If you needed to attach additional data to an Address (e.g. a UUID), you 
could use additional_data (addAdditionalData() in UUID).

As to the issue of having the same IP addresses in different data 
centers, e.g. link local addresses such as 192.168.x.x: the way it's 
done in the internet is with NATting, so the outgoing address is 
converted to a public, routable, address and the NAT's state table is 
updated with this information. The receiver would only see a public IP 
address. I guess you could do it similarly, but of course routing could 
be done in a hundred different ways. If you needed a routing hint, it 
could be tucked into UUID.additonal_data.


> I invite you to look at the code, you can download it here (including
> any files that were modified within JGroups):
> http://archive.jentfoo.com/coding_projects/treeMesh-V0.1.tar.gz
>
> This code is EXTREMELY young and untested. You will notice lots and
> lots of logging I have been using to debug this program. You will also
> notice TODO statements in places where I know I am doing things in a
> very inefficient way. Once all the bugs and kinks are worked out I plan
> to go back through and cleanup the code, as well as improve performance
> in a few places I just have not spent the time yet. For right now I
> have been trying to focus on technical correctness. But with that said,
> I am open to any suggestions you have, not just bug fixes, but code
> structure, etc.

I took a look, but I guess I'll wait for a more stable release. My focus 
right now is to help you resolve the changes you want in the JGroups 
core, so you can get your protocols to completion. I guess, reality is 
that I won't be able to look into your code in detail, but I can give 
you conceptual advice. For me, the code will be more or less a 
black-box, like a plugin, providing some cool functionality.:-)

> Once I feel the code is in a pretty solid state, and worth exposing 
> more I plan to do the following:
> * Update my original design document with more details and more
> specifics about the real implementation
> * Distribute the code for others to examine, and with it distribute the
> chat program I am making so that people can easily test it.

Yes. I guess people will only start looking into code if there is a cool 
application that they can run FIRST. Nobody (except CS students :-)) 
looks at code just for fun, but a lot of people could potentially use 
your stuff if you ship a cool application. If people start using your 
application, then some folks will also look at the code sooner or later, 
to fix bugs and/or add new features. This is what happened with JGroups, 
and I hope it'll happen with treemesh !
Cheers,


-- 
Bela Ban
Lead JGroups / Clustering Team
JBoss

------------------------------------------------------------------------------
Start uncovering the many advantages of virtual appliances
and start using them to simplify application deployment and
accelerate your shift to cloud computing.
http://p.sf.net/sfu/novell-sfdev2dev
_______________________________________________
Javagroups-development mailing list