Re: Advertisement presence component

Hi Paul,

My comments in your message.

Regards,

Fabrice

On Feb 17, 2006, at 5:31 AM, Poole, Paul P1 wrote:

> Hi Fabrice,
>
> Thanks for the response, I really appreciate you taking the time to  
> help me out.  I have added (inlined) some additional details below...
>
> Thanks,
>
> Paul
>
> From: Fabrice Di Meglio [mailto:[email protected]]
> Sent: Thursday, February 16, 2006 4:24 PM
> To: [email protected]
> Cc: Di Meglio Fabrice
> Subject: Re: [JXTA user] Advertisement presence component
>
> Hi Paul,
>
> I think we need more info about what you try to do to be able to  
> give you a good answer.
> [Poole, Paul] I am developing a framework around jxta that eases  
> development of jxta-based applications, increases overall quality  
> of service, and offers additional services that are not included in  
> jxta out of the box.  One of the services I would like to offer is  
> something I call "presence" awareness.

That is exactly what I have done with proXimio (you can freely  
download it at http://www.proximio.com).  Moreover proXimio deliver a  
framework for plugin any service around presence and can be embedded  
into any Java application. Current services are: 1-1 and Group Chat  
in TLS, File Transfer in TLS, Search on Metadata, Document Sharing,  
Voice over JXTA.

Daniel has also done something similar but focused only on presence.  
Look for his messages on the dev mailing list and http:// 
commons.jxta.org/

Last but not least, there is also the presence JXTA project which is  
a bit in "sleeping mode". I have been a owner since a couple of  
months but must confess that I could not have time for it. Now It's  
time to change all this and to reactivate it. Any volunteer for  
helping us will greatly welcome. First part should be defining the  
correct architecture so that this presence service is scalable.

> I would like to be able to offer to the end-developer a component  
> that allows them to monitor groups, nodes, files, etc that are  
> added to or removed (time out) from the network.  The use of this  
> component will be completely optional so that developers who are  
> developing an application for very large networks or extremely  
> bandwidth constrained networks can opt not to use this service.  I  
> am looking to develop something that is as scalable as possible.   
> For example, if the developer is building a chatting application,  
> they would be able to include the presence awareness component to  
> keep their network view updated.

Yep that's the idea
> So, let's say that a user has joined a group, they would be able to  
> view all the current members of the group and their list will  
> update in (mostly) realtime as members join and resign from the group.

Sure ;-)
>
> As stated into the wiki, such ADV repositories does not exist for  
> scalability. They will not able to scale to millions of devices/ 
> resources. You can look at the JXTA architecture papers that talk  
> about RDV, DHT, search and other items.
> [Poole, Paul] Yeah... I am concerned about scalability as well.  I  
> will try to make the component as scalable as possible.  That said,  
> it will be optional, so if it fails the scalability test for a  
> particular network, the end-developer can opt-out.
>
> The central idea is to have only caching of ADVs on edge Peers and  
> RDVs just cache indexes to ADVs (which is a position into the DHT).  
> When a Peer is searching for an ADV (whatever its type), he just  
> submit a query which may be forwarded by its RDV or another RDV in  
> the DHT to the real Peer(s) that can send a respnse. As the  
> response is ultimately sent by the edge peers, the query initiator  
> will get the latest version of the ADV and thus of its associated  
> resource.
> [Poole, Paul] It is good to know that the system ensures that the  
> latest advertisement is always sent.
>
> I have also noted that you talked about ADV deletion. This feature  
> is not part of JXTA platform currently. You can just publish an ADV  
> with some expiration / aging info. You may republish an ADV is you  
> need to change some of its parameters.
> [Poole, Paul] Yes, I have built the deletion capability by flushing  
> the old advertisement and re-publishing it with a 1 millisecond  
> timeout.  This is currently untested and I am not sure yet how well  
> this will work across the network.
My previous tests about using this for syncing presence status show  
that it is not a good way. Basically latency for ADV propagation is  
the main problem. When dealing with presence what do do not want is  
having delays into presence info propagation.
>
> As far as I understand your need, if your service will just have to  
> cope with a couple of hundred/thousands of "subscribers" your  
> proposal "2" may work depending on your OS or hardware.
> [Poole, Paul] In the end, my framework will be used to develop  
> projects that will range in size and useage considerably.  This is  
> not easily predictable, so I will have to offer flexible  
> alternatives - part of being in the research arena I guess :-)
>
> BTW, it is easier to put the bandwidth constraints on the edge  
> peers than on the RDVs/Relays peers. So your solution where Peers  
> submit a query/search at regular interval should be ok for  
> scalability.
> [Poole, Paul] This surprises me.  I would think that if all edge  
> peers are constantly sending out queries (maybe once a minute),  
> this would result in excessive bandwidth.
It is basically the same idea as BitTorrent versus a FTP server.
>
> Last comment, using PeerGroups and Peers partionning is fundamental  
> for global scalability.
> [Poole, Paul] What do you mean by "partitioning"?  How is this  
> fundamental for global scalability?  I am sorry... I just don't  
> understand this comment.

Partitioning means grouping Peers into several PeerGroups so that you  
dont have the case where everybody is in the same PeerGroup thus  
avoiding to put too much load on a RDV (I assume that the Peers are  
connected into a RDV in each PeerGroup) for ADV indexing/searching,  
pipes/routes resolutions... Depending on hardware/OS config, a normal  
RDV should be able to easily handle a couple of 1000 to 10,000 Peers  
but not 100,000. I remember Mike giving some data about this at last  
JXTA Kitchen but I dont have my notes with me.
>
> Hth,
>
> Fabrice
>
> On Feb 16, 2006, at 12:33 PM, Poole, Paul P1 wrote:
>
>> Hi all:
>>
>> I would like to solicit you all for some advice on creating an  
>> Advertisement aware system.  According to http://wiki.java.net/bin/ 
>> view/Jxta/PublishingDiscoveringPipeAndOtherAdvertiesements in the  
>> "How do peers know an advertisement has been published?" section,  
>> there is no system in place for listening for new advertisements  
>> or the expiration/deletion of others.  I was already aware of  
>> this, but was hoping that someone might have some ideas about how  
>> best to build this around jxta.  I would like to build an optional  
>> component that would allow developers to "opt-in" for  
>> notifications of changes in the network (additions and losses of  
>> peers, peergroups, etc).
>>
>> Are there any projects that are working on this?  Any thoughts on  
>> some more efficient ways to do this?
>>
>> 2 ideas I am kicking around right now are:
>>
>> 1. Each peer group uses the DiscoveryService to search every x  
>> minutes and compare the results with the current state.  I don't  
>> like this idea as I think it will be very resource intensive.   
>> Especially if there are many peergroups all utilizing the  
>> DiscoveryService to search… this will be a bandwidth bottleneck.
>>
>> 2. Peers that are rendezvous can monitor for changes in the  
>> advertisements (additions, deletions, etc).  Then changes can be  
>> messaged to a PeerGroup.  This way, only peers that want/need  
>> updates can subscribe to a particular peer group.  The messages  
>> can be throttled back to be more like digests every X seconds/ 
>> minutes/hours, etc or be allowed to be sent per change.
>>
>> I appreciate any thoughts/advice that anyone has to offer.
>>
>> Thanks,
>>
>>
>> Paul Poole
>> Software Engineer
>> Lockheed Martin
>>
>>
>