Re: [jgroups-dev] TreeMesh progress and questions

Replied inline

Bela Ban wrote:
> Hi Mike,
>
> I applied your changes and tried out treeMesh_NIO.xml with Draw (2 on 
> my Linux box and 2 on my Mac). The draw instances were able to find 
> each other, but the views shown never stabilized; meaning I never got 
> to 4 members. Also, sometimes a node would not be able to communicate 
> with the other nodes, for a few secs until the situation got cleared 
> up...
>
Hehe, you were not supposed to actually try to run it :-p

I know it is pretty broken still, I was just providing the full source 
code to look at my data structures (specifically MeshIdentifier), as 
well as the configuration options.
>
> Mike Jensen wrote:
>> I know it has been a while since you have heard from me. But I been
>> continuing to work on everything. At the bottom I have a link to
>> download the code for you to examine.
>>
>> I am in the process of creating a chat application that can run on top
>> of this. I have the basics of the chat application done, and it seems
>> to be working just fine on top of the TreeMesh. I still have other
>> things I want to do for it though:
>> * Implement meshGUI in the chat application, so that you can actually
>> see the connection structures that are being used for your chat session
>
> Good !
>
>> * Implement an IRC bridge, just to show an example of how a highly
>> strong coherency application can work together with something like what
>> I have built
>
> To interact with real IRC ! That'd be cool !
I got this built, and is about 90% working.  I don't know of any bugs 
atm, but I have not poked at it very hard.  I will start working on this 
again later, but for now I am focusing on the protocol again.
>
>> First a few notes about the implementation (so much I could write here,
>> I just outlined some bigger items):
>>
>> * It does look like I have to keep track of what messages we have seen.
>> Because messages are routed to us via multiple paths, I need to keep a
>> list of longs which represent random msgID's and ignore those we have
>> already seen. 
>
> Why random IDs ? Why not simply have the sender use monotonically 
> increasing IDs ?
>
> You could maintain the highest seqno received from each node, and 
> discard any message with a seqno which is lower than it. Whenever you 
> receive a higher seqno, update that value in the hashmap.
>
> This of course will lead to message loss in certain cases, e.g. if you 
> receive messages 5 8 6 5 7 9 from P, you'll only deliver to the 
> application messages 5, 8 and 9 (and discard 6, 5 and 7). However, 
> this should not be such a big issue as you're mainly using TCP as 
> transport, which guarantees ordering anyway.
>
> So keeping the highest seqnos seems to be a very simple and efficient 
> way of weeding out duplicates.
>
This is a good idea.  My co-worker here recommended that I could maybe 
do this, and have a sliding window of ID's.  That way we can even get 
out of order messages that are within the sliding window.  The goal here 
would be that the window could be much smaller than my existing list.  
But for the short term I may just implement it as you described, in the 
hopes that this event should be fairly rare with TCP.
>
> <SNIP>
>
>> * I have created a new MeshView which extends View. So far this seems
>> to work just fine as long as the application is aware that it could be
>> getting a MeshView instead of a View (assuming the application wants to
>> take advantage of the mesh structures).
>
> Hmm. Another way of doing this is to use the payload in View. Well, 
> actually I've deprecated use of the payload, but if there is a real 
> need, we can reinstate it... This would obliviate the need for 
> MeshView. Well, actually, maybe I like MeshView more after all, so I 
> can go forward with deprecating payload !
>
Yes, I looked at the payload.  But I think it is fine to depreciate it.  
I have built the chat application, and honestly I think having a class 
just extend view probably works just fine (you just may need to do 
instanceof checks if you want your app to work with different protocols 
and still take advantage of knowing who you are directly connected to).  
But all the View implementations still work fine with MeshView (AFAIK).
>> Although I can't seem to get the unicast protocol to work with my 
>> tree protocol, and I suspect the
>> issue may be because of this. But I have spent almost no time to 
>> diagnose the problem.
> UNICAST / UNICAST2 should work without views anyway ! Although it 
> would be good to pass a view to them every now and then, so they can 
> clean out connections to / from old members. Note that UNICAST2 
> doesn't use any acks.
>
I still have not looked into this yet.
>
> <SNIP>
>
>> And now for the question!
>>
>> My biggest issue is that I have extended UUID with a new class
>> "MeshIdentifier" that I want to function as the Address. The problem is
>> that despite this trying to mimic an Address as close as I can figure
>> out, I can't seem to get messages to actually route with this address.
>> Further more, I need to figure out how to get this Address to be
>> integrated deeper into JGroups so that when you call
>> JChannel.getLocalAddress() it returns this UUID form. The reason this
>> is important is because I want Addresses to be unique per instance, so
>> that if machines span multiple datacenters than we are able to uniquely
>> identify them even if they have the same IP/port. Another example of
>> such a situation would be running multiple instances of the process on
>> the same machine. Suggestions on how to better handle this? I invite
>> you to look at my implementation, but right now MeshIdentifier just
>> stores the real address + the super class UUID, and i just call
>> .getAddress() before I route any messages. This is less than ideal as
>> it means that I may or may not be sending the message to the node I
>> think I am.
>
> Not sure I understand what you're trying to achieve here. 
I have been trying to work on this...but not having much luck.  Maybe I 
am misunderstanding.

The way I read your response is try to use an Address, then set the UUID 
as additional in the addresses (I assume that this information would be 
transmitted on the wire as well when an address is sent).  So when I try 
to route a message or receive a message, I check which node I am 
referencing to, not just by the Address, but verifying that the 
AdditionalData matches as well.

So basically the opposite of what I am trying to do currently.  Where I 
am currently storing an Address inside a UUID, I would be storing a UUID 
inside an Address.

The problem is that I don't see any getter or setters for additional 
data in Address....but based on your responses, maybe I was getting 
close and you can help me make my current implementation (UUID that 
stores an Address) work.
> A UUID maps to a PhysicalAddress (IP address) and is always unique 
> over space and time, also across different data centers. When we send 
> a message to a UUID, we look up the physical address associated with 
> that UUID and grab the connection to that physical address (destination).
>
> The mapping between UUIDs and physical addresses is returned by the 
> discovery protocol (TCPPING in your example).
I think what you said above may be the key.  Let me describe where these 
UUID's are currently coming from.

Right know at construction time of TREEMESH we create a new random UUID 
( line 81 of TREEMESH.java ).  This is the UUID I have been using.  When 
we connect to a remote node, we send them THIS UUID that was constructed 
earlier (line 686 is constructing the handshake, it is read in line 
493).  The hope was that they could then communicate using the mesh 
identifier that extends this UUID (but by using the PhysicalAddress 
stored inside it).

What is happening if we try this, is that we end up connecting to one of 
our initial discovery nodes (via the address given from 
Event.FIND_INITIAL_MBRS).  After connected we exchange our handshake 
information which contains this UUID we constructed at initialization.  
We then try to use this for any further communication.  But then once 
connected the node will timeout because it tries to broadcast its 
heartbeat, but the remote node never hears it because any messages that 
are being sent with an input destination of type MeshIdentifier never 
arrive.

Further more, if a node was to ever learn about another node from other 
messages, and then wants to establish a leaf connection by sending a 
message with MeshIdentifier, it never arrives and the connection is 
never established.

This was the main reason I had included my code, so you can examine 
MeshIdentifier.  I did not expect the rest of the JGroups system to 
understand or be aware of this UUID.  What I was figuring was actually 
something very similar as to what you suggested.  Have the Address 
inside the MeshIdentifier be replaced with possibly a public Address, 
and use the UUID that it extends to reference the specific node.  So I 
was expecting JGroups to use the JGroups address that is stored inside 
the MeshIdentifier.  Which is why MeshIdentifier implements 
PhysicalAddress, it uses the internal stored Address to fulfill the 
interface.

So....that is the road I was going down, and why I thought it would 
work.  But it does not seem to be able to actually send messages (I am 
guessing the fact that it extends a UUID is confusing something, and 
thus it never actually recognizes that it implements PhysicalAddress).

Based on your suggestion, is this what I need to do:
Instead of using a UUID that I construct at initialization, should I get 
this UUID from somewhere else?  If I get this UUID from somewhere else 
on the local node, what about the remote nodes.  Should I still include 
the UUID in the handshake data so that we know exactly what node we are 
talking to, or should I get this UUID from some other part of the system?

If I get the correct UUID, and still use MeshIdentifier to store it, do 
I even need additional_data for anything?  Meaning, if I were to use a 
UUID that the rest of the system understands, can I still use the 
physical address stored in MeshIdentifier to determine the message 
routing address, and the UUID to represent the actual node?

Given that, what suggestions do you have?  Sorry if that is confusing, 
or sounds like I was repeating myself, just trying to very clearly 
indicate what I mean.
>
> If you needed to attach additional data to an Address (e.g. a UUID), 
> you could use additional_data (addAdditionalData() in UUID).
>
> As to the issue of having the same IP addresses in different data 
> centers, e.g. link local addresses such as 192.168.x.x: the way it's 
> done in the internet is with NATting, so the outgoing address is 
> converted to a public, routable, address and the NAT's state table is 
> updated with this information. The receiver would only see a public IP 
> address. I guess you could do it similarly, but of course routing 
> could be done in a hundred different ways. If you needed a routing 
> hint, it could be tucked into UUID.additonal_data.
>
>> I invite you to look at the code, you can download it here (including
>> any files that were modified within JGroups):
>> http://archive.jentfoo.com/coding_projects/treeMesh-V0.1.tar.gz
>>
>> This code is EXTREMELY young and untested. You will notice lots and
>> lots of logging I have been using to debug this program. You will also
>> notice TODO statements in places where I know I am doing things in a
>> very inefficient way. Once all the bugs and kinks are worked out I plan
>> to go back through and cleanup the code, as well as improve performance
>> in a few places I just have not spent the time yet. For right now I
>> have been trying to focus on technical correctness. But with that said,
>> I am open to any suggestions you have, not just bug fixes, but code
>> structure, etc.
>
> I took a look, but I guess I'll wait for a more stable release. My 
> focus right now is to help you resolve the changes you want in the 
> JGroups core, so you can get your protocols to completion. I guess, 
> reality is that I won't be able to look into your code in detail, but 
> I can give you conceptual advice. For me, the code will be more or 
> less a black-box, like a plugin, providing some cool functionality.:-)
>
That sounds just fine for me.  I don't mind if you never look at the 
code, I just feel that including it for reference is sometimes easier.  
For me I like to reference the code while providing help.  But I by no 
means have any expectations that it needs to be examined, just offering 
as a curtsy.
> <SNIP>

Thanks for your help!

------------------------------------------------------------------------------
Start uncovering the many advantages of virtual appliances
and start using them to simplify application deployment and
accelerate your shift to cloud computing.
http://p.sf.net/sfu/novell-sfdev2dev
_______________________________________________
Javagroups-development mailing list