[jgroups-dev] provided services bug details

I may start to look at this my previous email today (I have been working 
on other issues).  Specifically dealing with an issue where two graphs 
could become split and not merge.  Although I think I have that solved now.

I am just emailing because I think I found a bug.  Previously I had said 
that when I use "requiredDownService" it would complain that no up 
service provided it.  I have figured out where the bug here lies.

In stack.Configurator.java line 484 (CVS was updated yesterday), the 
text is wrong.  It says that "not provided by any layers above" but 
should read "not provided by any layers below".

And the second part of the bug is that Discovery is not reporting all 
the services it has available.  At line 160 in Discovery.java the 
following should be added:
ret.addElement(new Integer(Event.FIND_ALL_MBRS));

Should GET_PHYSICAL_ADDRESS be listed here as well?

Those two changes have fixed my issue.  I discovered this a while ago 
and had mentioned it.  But I only recently found the true problem since 
I have been working on this split graph issue.

Let me know if you have any questions.  Thanks

Mike Jensen wrote:
> Ah....okay, things are making sense.  I have made good progress today 
> (running lots better after fixing lots of small bugs).
>
> I am casting almost all addresses now to UUID's for safety (exceptions 
> include things like FIND_INITIAL_MBRS).  And things seem to be working 
> much better now.  But I have a weird situation in connecting I am not 
> sure how to solve.  Basically the situation is like this:
>
> 1) We get an up event of MeshHandshakeHeader from a node wishing to 
> establish a connection to us
> 2) We read the information they provided, all is good, and we decide 
> to accept the connection.  So we then try to reply with our own 
> information for the connecting node.
> 3) In psudo code it looks like this:
>
> if (hdr instanceof MeshHandshakeHeader) {
>    newConnectionAttempt((String)msg.getObject(), (UUID)msg.getSrc(), 
> <boolean::IsResponse>);
> }
>
> newConnectionAttempt(String data, UUID source, boolean isResponse) {
>    // parse data
>    // check if we want to keep this connection
>    if (keepConnection && !isResponse) {
>       // send response
>       Message msg = new Message(source, local_addr, 
> createHandshakeData(conType));
>       msg.putHeaderIfAbsent(getID(), new MeshHandshakeHeader(true));
>       down_prot.down(new Event(Event.MSG, msg));
>    }
> }
>
> The issue is that even though I am using the UUID that was the source 
> of the incoming connection request, when I try to send using this UUID 
> as the destination I get the following:
>
> WARNING: <local hostname>: no physical address for <UUID of remote 
> node>, dropping message
>
> Why would it be unable to look up the UUID of the source of a message 
> we just received?  What ends up happening is that the connection 
> attempt ends up timing out, since the response was never received.  
> After the connection times out, the node will try to establish another 
> connection.  Then on the second try it ALWAYS works, it works if it is 
> connecting to the same node, or a different node.  Why would it always 
> work on the second try but not the first when the code paths are the 
> same?
>
> As for having another demo/phone call.  That sounds good, I am always 
> willing to talk and explain where I am.  I just am trying to be 
> sensitive to your time.  Now that I am making progress again, I don't 
> feel that there is an urgent need for this.  But maybe we will want to 
> do this later this week (like maybe Thursday morning for me).
>
> Thanks for your help.
>
> Bela Ban wrote:
>> Hey Mike
>>
>> Mike Jensen wrote:
>>>>> * Implement an IRC bridge, just to show an example of how a 
>>>>> highlystrong coherency application can work together with 
>>>>> something like what
>>>>> I have built
>>>> To interact with real IRC ! That'd be cool !
>>> I got this built, and is about 90% working. I don't know of any bugs
>>> atm, but I have not poked at it very hard. I will start working on this
>>> again later, but for now I am focusing on the protocol again.
>>
>> OK. Why don't you post a link to it here, once it's in a presentable 
>> state.
>>
>>
>>>>> * It does look like I have to keep track of what messages we have 
>>>>> seen.
>>>>> Because messages are routed to us via multiple paths, I need to 
>>>>> keep a
>>>>> list of longs which represent random msgID's and ignore those we have
>>>>> already seen.
>>>> Why random IDs ? Why not simply have the sender use monotonically  
>>>> increasing IDs ?
>>>>
>>>> You could maintain the highest seqno received from each node, and
>>>> discard any message with a seqno which is lower than it. Whenever you
>>>> receive a higher seqno, update that value in the hashmap.
>>>>
>>>> This of course will lead to message loss in certain cases, e.g. if you
>>>> receive messages 5 8 6 5 7 9 from P, you'll only deliver to the
>>>> application messages 5, 8 and 9 (and discard 6, 5 and 7). However,
>>>> this should not be such a big issue as you're mainly using TCP as
>>>> transport, which guarantees ordering anyway.
>>>>
>>>> So keeping the highest seqnos seems to be a very simple and efficient
>>>> way of weeding out duplicates.
>>>>
>>> This is a good idea. My co-worker here recommended that I could maybe
>>> do this, and have a sliding window of ID's. That way we can even get
>>> out of order messages that are within the sliding window. The goal here
>>> would be that the window could be much smaller than my existing list.
>>> But for the short term I may just implement it as you described, in the
>>> hopes that this event should be fairly rare with TCP.
>>
>> A sliding window *without* retransmission I asssume ? Just like UDP 
>> datagrams are assembled by collecting fragmented IP packets, no 
>> retransmission, and if we don't have all IP fragments within a 
>> certain time, we drop the entire datagram...
>>
>> Your timeout should probably be very small though, as you mentioned 
>> you'd rather trash a packet than waiting for a missing packet...
>>
>> I guess, for now, this point is pretty moot if you run over TCP. 
>> Should you decide to run over a datagram based transport though, you 
>> need to reconsider this.
>>
>>
>>
>>>>> * I have created a new MeshView which extends View. So far this 
>>>>> seems to work just fine as long as the application is aware that 
>>>>> it could be
>>>>> getting a MeshView instead of a View (assuming the application 
>>>>> wants to take advantage of the mesh structures).
>>>> Hmm. Another way of doing this is to use the payload in View. Well,
>>>> actually I've deprecated use of the payload, but if there is a real
>>>> need, we can reinstate it... This would obliviate the need for
>>>> MeshView. Well, actually, maybe I like MeshView more after all, so I
>>>> can go forward with deprecating payload !
>>>>
>>> Yes, I looked at the payload. But I think it is fine to depreciate it.
>>> I have built the chat application, and honestly I think having a class
>>> just extend view probably works just fine (you just may need to do
>>> instanceof checks if you want your app to work with different protocols
>>> and still take advantage of knowing who you are directly connected to).
>>> But all the View implementations still work fine with MeshView (AFAIK).
>>
>> OK, agreed.
>>
>>
>>> The way I read your response is try to use an Address, then set the 
>>> UUID as additional in the addresses
>> I suggested to use a UUID and use its additional_data field to carry 
>> the information provided by MeshIdentifier.
>>
>>> (I assume that this information would be  transmitted on the wire as 
>>> well when an address is sent).
>>
>> Yes, exactly. So you'd have the additional data associated with a 
>> UUID available everywhere. I'm actually reconsidering my intent to 
>> deprecate and remove additional_data; as a matter of fact, I might 
>> even add methods to add/remove/get additional data to Address itself...
>>
>>> So when I try to route a message or receive a message, I check which 
>>> node I am
>>> referencing to, not just by the Address, but verifying that the 
>>> AdditionalData matches as well.
>>
>> Well, for comparison and equality, the address (UUID) would be all 
>> you need. For routing, you'd need to access the additional info, yes.
>>
>>> So basically the opposite of what I am trying to do currently. Where 
>>> I am currently storing an Address inside a UUID, I would be storing 
>>> a UUID
>>> inside an Address.
>>
>> No, you'd be storing additional data in a UUID (Address) !
>>
>>> The problem is that I don't see any getter or setters for additional
>>> data in Address....but based on your responses, maybe I was getting
>>> close and you can help me make my current implementation (UUID that
>>> stores an Address) work.
>>
>> Yep - you'd have to narrow the Address down to a UUID, probably not a 
>> good idea should we ever change the address to be something other 
>> than UUID... Currently, use an instanceof check (should always be true).
>>
>> I might actually propagate the getters and setters to Address itself 
>> (see above), let's see whether I'm stumbling across more use cases 
>> like yours. People have been bugging me forever to do this... :-)
>>
>>
>>>> A UUID maps to a PhysicalAddress (IP address) and is always unique
>>>> over space and time, also across different data centers. When we send
>>>> a message to a UUID, we look up the physical address associated with
>>>> that UUID and grab the connection to that physical address 
>>>> (destination).
>>>>
>>>> The mapping between UUIDs and physical addresses is returned by the
>>>> discovery protocol (TCPPING in your example).
>>> I think what you said above may be the key. Let me describe where these
>>> UUID's are currently coming from.
>>>
>>> Right know at construction time of TREEMESH we create a new random UUID
>>> ( line 81 of TREEMESH.java ). This is the UUID I have been using. When
>>> we connect to a remote node, we send them THIS UUID that was 
>>> constructed
>>> earlier (line 686 is constructing the handshake, it is read in line
>>> 493). The hope was that they could then communicate using the mesh
>>> identifier that extends this UUID (but by using the PhysicalAddress
>>> stored inside it).
>>
>> Why don't you use the UUID that was created as part of the JChannel ? 
>> JChannel.getAddress() would return it...
>>
>>> What is happening if we try this, is that we end up connecting to 
>>> one of
>>> our initial discovery nodes (via the address given from
>>> Event.FIND_INITIAL_MBRS). After connected we exchange our handshake
>>> information which contains this UUID we constructed at initialization.
>>> We then try to use this for any further communication. But then once
>>> connected the node will timeout because it tries to broadcast its
>>> heartbeat, but the remote node never hears it because any messages that
>>> are being sent with an input destination of type MeshIdentifier never
>>> arrive.
>>>
>>> Further more, if a node was to ever learn about another node from other
>>> messages, and then wants to establish a leaf connection by sending a
>>> message with MeshIdentifier, it never arrives and the connection is
>>> never established.
>>
>> I guess the transport (TP) doesn't know about the association between 
>> your UUID and the channel's UUID with a physical address. If you use 
>> JChannel.local_addr, then this should work.
>>
>>> This was the main reason I had included my code, so you can examine
>>> MeshIdentifier. I did not expect the rest of the JGroups system to
>>> understand or be aware of this UUID. What I was figuring was actually
>>> something very similar as to what you suggested. Have the Address
>>> inside the MeshIdentifier be replaced with possibly a public Address,
>>> and use the UUID that it extends to reference the specific node. So I
>>> was expecting JGroups to use the JGroups address that is stored inside
>>> the MeshIdentifier. Which is why MeshIdentifier implements
>>> PhysicalAddress, it uses the internal stored Address to fulfill the
>>> interface.
>>>
>>> So....that is the road I was going down, and why I thought it would
>>> work. But it does not seem to be able to actually send messages (I am
>>> guessing the fact that it extends a UUID is confusing something, and
>>> thus it never actually recognizes that it implements PhysicalAddress).
>>>
>>> Based on your suggestion, is this what I need to do:
>>> Instead of using a UUID that I construct at initialization, should I 
>>> get
>>> this UUID from somewhere else? 
>>
>> Yes, from the JChannel
>>
>>> If I get this UUID from somewhere else
>>> on the local node, what about the remote nodes. Should I still include
>>> the UUID in the handshake data so that we know exactly what node we are
>>> talking to, or should I get this UUID from some other part of the 
>>> system?
>>
>> You shouldn't need to do this; the discovery protocols take care of 
>> disseminating the UUID-PhysicalAddress mappings. The UUIDs themselves 
>> are known as soon as peer receives them.
>>
>>> If I get the correct UUID, and still use MeshIdentifier to store it, do
>>> I even need additional_data for anything? Meaning, if I were to use a
>>> UUID that the rest of the system understands, can I still use the
>>> physical address stored in MeshIdentifier to determine the message
>>> routing address, and the UUID to represent the actual node?
>>
>> Yes - provided the physical address is routable, e.g. if you have 
>> cluster islands A and B, and both have IP addresses in the 
>> 192.168.5.0/24 network, then sending a message from island A to B 
>> will fail, as the routing layer assume any 192.168.5.* is local ! 
>> That's where IMO the routing intelligence ca use additional_data to 
>> perform correct routing. E.g. this data could contain the address of 
>> the nearest gateway to which to send messages. This gateway would be 
>> a public IP in B...
>>
>>> Given that, what suggestions do you have? Sorry if that is confusing,
>>> or sounds like I was repeating myself, just trying to very clearly
>>> indicate what I mean.
>>
>> Maybe we should have another talk... WDYT ?
>>
>>
>

------------------------------------------------------------------------------
Nokia and AT&T present the 2010 Calling All Innovators-North America contest
Create new apps & games for the Nokia N8 for consumers in  U.S. and Canada
$10 million total in prizes - $4M cash, 500 devices, nearly $6M in marketing
Develop with Nokia Qt SDK, Web Runtime, or Java and Publish to Ovi Store 
http://p.sf.net/sfu/nokia-dev2dev
_______________________________________________
Javagroups-development mailing list