Re: [jgroups-dev] provided services bug details
Bela Ban <[email protected]>
| Newsgroups | gmane.comp.java.javagroups.devel |
|---|---|
| Message-ID | <[email protected]> |
Thanks Mike !
Fixed in 2.11 and 2.10.1.
Re your other email: I'll get back to you soon, I'm busy preparing my
talk for JUDCon...
Mike Jensen wrote:
> I may start to look at this my previous email today (I have been working
> on other issues). Specifically dealing with an issue where two graphs
> could become split and not merge. Although I think I have that solved now.
>
> I am just emailing because I think I found a bug. Previously I had said
> that when I use "requiredDownService" it would complain that no up
> service provided it. I have figured out where the bug here lies.
>
> In stack.Configurator.java line 484 (CVS was updated yesterday), the
> text is wrong. It says that "not provided by any layers above" but
> should read "not provided by any layers below".
>
> And the second part of the bug is that Discovery is not reporting all
> the services it has available. At line 160 in Discovery.java the
> following should be added:
> ret.addElement(new Integer(Event.FIND_ALL_MBRS));
>
> Should GET_PHYSICAL_ADDRESS be listed here as well?
>
> Those two changes have fixed my issue. I discovered this a while ago
> and had mentioned it. But I only recently found the true problem since
> I have been working on this split graph issue.
>
> Let me know if you have any questions. Thanks
>
> Mike Jensen wrote:
>
>> Ah....okay, things are making sense. I have made good progress today
>> (running lots better after fixing lots of small bugs).
>>
>> I am casting almost all addresses now to UUID's for safety (exceptions
>> include things like FIND_INITIAL_MBRS). And things seem to be working
>> much better now. But I have a weird situation in connecting I am not
>> sure how to solve. Basically the situation is like this:
>>
>> 1) We get an up event of MeshHandshakeHeader from a node wishing to
>> establish a connection to us
>> 2) We read the information they provided, all is good, and we decide
>> to accept the connection. So we then try to reply with our own
>> information for the connecting node.
>> 3) In psudo code it looks like this:
>>
>> if (hdr instanceof MeshHandshakeHeader) {
>> newConnectionAttempt((String)msg.getObject(), (UUID)msg.getSrc(),
>> <boolean::IsResponse>);
>> }
>>
>> newConnectionAttempt(String data, UUID source, boolean isResponse) {
>> // parse data
>> // check if we want to keep this connection
>> if (keepConnection && !isResponse) {
>> // send response
>> Message msg = new Message(source, local_addr,
>> createHandshakeData(conType));
>> msg.putHeaderIfAbsent(getID(), new MeshHandshakeHeader(true));
>> down_prot.down(new Event(Event.MSG, msg));
>> }
>> }
>>
>> The issue is that even though I am using the UUID that was the source
>> of the incoming connection request, when I try to send using this UUID
>> as the destination I get the following:
>>
>> WARNING: <local hostname>: no physical address for <UUID of remote
>> node>, dropping message
>>
>> Why would it be unable to look up the UUID of the source of a message
>> we just received? What ends up happening is that the connection
>> attempt ends up timing out, since the response was never received.
>> After the connection times out, the node will try to establish another
>> connection. Then on the second try it ALWAYS works, it works if it is
>> connecting to the same node, or a different node. Why would it always
>> work on the second try but not the first when the code paths are the
>> same?
>>
>> As for having another demo/phone call. That sounds good, I am always
>> willing to talk and explain where I am. I just am trying to be
>> sensitive to your time. Now that I am making progress again, I don't
>> feel that there is an urgent need for this. But maybe we will want to
>> do this later this week (like maybe Thursday morning for me).
>>
>> Thanks for your help.
>>
>> Bela Ban wrote:
>>
>>> Hey Mike
>>>
>>> Mike Jensen wrote:
>>>
>>>>>> * Implement an IRC bridge, just to show an example of how a
>>>>>> highlystrong coherency application can work together with
>>>>>> something like what
>>>>>> I have built
>>>>>>
>>>>> To interact with real IRC ! That'd be cool !
>>>>>
>>>> I got this built, and is about 90% working. I don't know of any bugs
>>>> atm, but I have not poked at it very hard. I will start working on this
>>>> again later, but for now I am focusing on the protocol again.
>>>>
>>> OK. Why don't you post a link to it here, once it's in a presentable
>>> state.
>>>
>>>
>>>
>>>>>> * It does look like I have to keep track of what messages we have
>>>>>> seen.
>>>>>> Because messages are routed to us via multiple paths, I need to
>>>>>> keep a
>>>>>> list of longs which represent random msgID's and ignore those we have
>>>>>> already seen.
>>>>>>
>>>>> Why random IDs ? Why not simply have the sender use monotonically
>>>>> increasing IDs ?
>>>>>
>>>>> You could maintain the highest seqno received from each node, and
>>>>> discard any message with a seqno which is lower than it. Whenever you
>>>>> receive a higher seqno, update that value in the hashmap.
>>>>>
>>>>> This of course will lead to message loss in certain cases, e.g. if you
>>>>> receive messages 5 8 6 5 7 9 from P, you'll only deliver to the
>>>>> application messages 5, 8 and 9 (and discard 6, 5 and 7). However,
>>>>> this should not be such a big issue as you're mainly using TCP as
>>>>> transport, which guarantees ordering anyway.
>>>>>
>>>>> So keeping the highest seqnos seems to be a very simple and efficient
>>>>> way of weeding out duplicates.
>>>>>
>>>>>
>>>> This is a good idea. My co-worker here recommended that I could maybe
>>>> do this, and have a sliding window of ID's. That way we can even get
>>>> out of order messages that are within the sliding window. The goal here
>>>> would be that the window could be much smaller than my existing list.
>>>> But for the short term I may just implement it as you described, in the
>>>> hopes that this event should be fairly rare with TCP.
>>>>
>>> A sliding window *without* retransmission I asssume ? Just like UDP
>>> datagrams are assembled by collecting fragmented IP packets, no
>>> retransmission, and if we don't have all IP fragments within a
>>> certain time, we drop the entire datagram...
>>>
>>> Your timeout should probably be very small though, as you mentioned
>>> you'd rather trash a packet than waiting for a missing packet...
>>>
>>> I guess, for now, this point is pretty moot if you run over TCP.
>>> Should you decide to run over a datagram based transport though, you
>>> need to reconsider this.
>>>
>>>
>>>
>>>
>>>>>> * I have created a new MeshView which extends View. So far this
>>>>>> seems to work just fine as long as the application is aware that
>>>>>> it could be
>>>>>> getting a MeshView instead of a View (assuming the application
>>>>>> wants to take advantage of the mesh structures).
>>>>>>
>>>>> Hmm. Another way of doing this is to use the payload in View. Well,
>>>>> actually I've deprecated use of the payload, but if there is a real
>>>>> need, we can reinstate it... This would obliviate the need for
>>>>> MeshView. Well, actually, maybe I like MeshView more after all, so I
>>>>> can go forward with deprecating payload !
>>>>>
>>>>>
>>>> Yes, I looked at the payload. But I think it is fine to depreciate it.
>>>> I have built the chat application, and honestly I think having a class
>>>> just extend view probably works just fine (you just may need to do
>>>> instanceof checks if you want your app to work with different protocols
>>>> and still take advantage of knowing who you are directly connected to).
>>>> But all the View implementations still work fine with MeshView (AFAIK).
>>>>
>>> OK, agreed.
>>>
>>>
>>>
>>>> The way I read your response is try to use an Address, then set the
>>>> UUID as additional in the addresses
>>>>
>>> I suggested to use a UUID and use its additional_data field to carry
>>> the information provided by MeshIdentifier.
>>>
>>>
>>>> (I assume that this information would be transmitted on the wire as
>>>> well when an address is sent).
>>>>
>>> Yes, exactly. So you'd have the additional data associated with a
>>> UUID available everywhere. I'm actually reconsidering my intent to
>>> deprecate and remove additional_data; as a matter of fact, I might
>>> even add methods to add/remove/get additional data to Address itself...
>>>
>>>
>>>> So when I try to route a message or receive a message, I check which
>>>> node I am
>>>> referencing to, not just by the Address, but verifying that the
>>>> AdditionalData matches as well.
>>>>
>>> Well, for comparison and equality, the address (UUID) would be all
>>> you need. For routing, you'd need to access the additional info, yes.
>>>
>>>
>>>> So basically the opposite of what I am trying to do currently. Where
>>>> I am currently storing an Address inside a UUID, I would be storing
>>>> a UUID
>>>> inside an Address.
>>>>
>>> No, you'd be storing additional data in a UUID (Address) !
>>>
>>>
>>>> The problem is that I don't see any getter or setters for additional
>>>> data in Address....but based on your responses, maybe I was getting
>>>> close and you can help me make my current implementation (UUID that
>>>> stores an Address) work.
>>>>
>>> Yep - you'd have to narrow the Address down to a UUID, probably not a
>>> good idea should we ever change the address to be something other
>>> than UUID... Currently, use an instanceof check (should always be true).
>>>
>>> I might actually propagate the getters and setters to Address itself
>>> (see above), let's see whether I'm stumbling across more use cases
>>> like yours. People have been bugging me forever to do this... :-)
>>>
>>>
>>>
>>>>> A UUID maps to a PhysicalAddress (IP address) and is always unique
>>>>> over space and time, also across different data centers. When we send
>>>>> a message to a UUID, we look up the physical address associated with
>>>>> that UUID and grab the connection to that physical address
>>>>> (destination).
>>>>>
>>>>> The mapping between UUIDs and physical addresses is returned by the
>>>>> discovery protocol (TCPPING in your example).
>>>>>
>>>> I think what you said above may be the key. Let me describe where these
>>>> UUID's are currently coming from.
>>>>
>>>> Right know at construction time of TREEMESH we create a new random UUID
>>>> ( line 81 of TREEMESH.java ). This is the UUID I have been using. When
>>>> we connect to a remote node, we send them THIS UUID that was
>>>> constructed
>>>> earlier (line 686 is constructing the handshake, it is read in line
>>>> 493). The hope was that they could then communicate using the mesh
>>>> identifier that extends this UUID (but by using the PhysicalAddress
>>>> stored inside it).
>>>>
>>> Why don't you use the UUID that was created as part of the JChannel ?
>>> JChannel.getAddress() would return it...
>>>
>>>
>>>> What is happening if we try this, is that we end up connecting to
>>>> one of
>>>> our initial discovery nodes (via the address given from
>>>> Event.FIND_INITIAL_MBRS). After connected we exchange our handshake
>>>> information which contains this UUID we constructed at initialization.
>>>> We then try to use this for any further communication. But then once
>>>> connected the node will timeout because it tries to broadcast its
>>>> heartbeat, but the remote node never hears it because any messages that
>>>> are being sent with an input destination of type MeshIdentifier never
>>>> arrive.
>>>>
>>>> Further more, if a node was to ever learn about another node from other
>>>> messages, and then wants to establish a leaf connection by sending a
>>>> message with MeshIdentifier, it never arrives and the connection is
>>>> never established.
>>>>
>>> I guess the transport (TP) doesn't know about the association between
>>> your UUID and the channel's UUID with a physical address. If you use
>>> JChannel.local_addr, then this should work.
>>>
>>>
>>>> This was the main reason I had included my code, so you can examine
>>>> MeshIdentifier. I did not expect the rest of the JGroups system to
>>>> understand or be aware of this UUID. What I was figuring was actually
>>>> something very similar as to what you suggested. Have the Address
>>>> inside the MeshIdentifier be replaced with possibly a public Address,
>>>> and use the UUID that it extends to reference the specific node. So I
>>>> was expecting JGroups to use the JGroups address that is stored inside
>>>> the MeshIdentifier. Which is why MeshIdentifier implements
>>>> PhysicalAddress, it uses the internal stored Address to fulfill the
>>>> interface.
>>>>
>>>> So....that is the road I was going down, and why I thought it would
>>>> work. But it does not seem to be able to actually send messages (I am
>>>> guessing the fact that it extends a UUID is confusing something, and
>>>> thus it never actually recognizes that it implements PhysicalAddress).
>>>>
>>>> Based on your suggestion, is this what I need to do:
>>>> Instead of using a UUID that I construct at initialization, should I
>>>> get
>>>> this UUID from somewhere else?
>>>>
>>> Yes, from the JChannel
>>>
>>>
>>>> If I get this UUID from somewhere else
>>>> on the local node, what about the remote nodes. Should I still include
>>>> the UUID in the handshake data so that we know exactly what node we are
>>>> talking to, or should I get this UUID from some other part of the
>>>> system?
>>>>
>>> You shouldn't need to do this; the discovery protocols take care of
>>> disseminating the UUID-PhysicalAddress mappings. The UUIDs themselves
>>> are known as soon as peer receives them.
>>>
>>>
>>>> If I get the correct UUID, and still use MeshIdentifier to store it, do
>>>> I even need additional_data for anything? Meaning, if I were to use a
>>>> UUID that the rest of the system understands, can I still use the
>>>> physical address stored in MeshIdentifier to determine the message
>>>> routing address, and the UUID to represent the actual node?
>>>>
>>> Yes - provided the physical address is routable, e.g. if you have
>>> cluster islands A and B, and both have IP addresses in the
>>> 192.168.5.0/24 network, then sending a message from island A to B
>>> will fail, as the routing layer assume any 192.168.5.* is local !
>>> That's where IMO the routing intelligence ca use additional_data to
>>> perform correct routing. E.g. this data could contain the address of
>>> the nearest gateway to which to send messages. This gateway would be
>>> a public IP in B...
>>>
>>>
>>>> Given that, what suggestions do you have? Sorry if that is confusing,
>>>> or sounds like I was repeating myself, just trying to very clearly
>>>> indicate what I mean.
>>>>
>>> Maybe we should have another talk... WDYT ?
>>>
>>>
>>>
>
> ------------------------------------------------------------------------------
> Nokia and AT&T present the 2010 Calling All Innovators-North America contest
> Create new apps & games for the Nokia N8 for consumers in U.S. and Canada
> $10 million total in prizes - $4M cash, 500 devices, nearly $6M in marketing
> Develop with Nokia Qt SDK, Web Runtime, or Java and Publish to Ovi Store
> http://p.sf.net/sfu/nokia-dev2dev
> _______________________________________________
> Javagroups-development mailing list
>
>
>
--
Bela Ban
Lead JGroups / Clustering Team
JBoss
------------------------------------------------------------------------------
Nokia and AT&T present the 2010 Calling All Innovators-North America contest
Create new apps & games for the Nokia N8 for consumers in U.S. and Canada
$10 million total in prizes - $4M cash, 500 devices, nearly $6M in marketing
Develop with Nokia Qt SDK, Web Runtime, or Java and Publish to Ovi Store
http://p.sf.net/sfu/nokia-dev2dev
_______________________________________________
Javagroups-development mailing list