Re: Using the DHT to find download sources

--- In [email protected], Michael Rogers <m.rogers@...> wrote:
>
> Arne Babenhauserheide wrote:
> >> I agree with "how do you find out before searching that your search
> >> will be a rare one?". This is a problem we need to solve. :)
> > 
> > Maybe it would be possible to always do a Gnutella search first,
and if that 
> > generates less than 300 results, fire up a Kad search afterwards.
> 
> Another possibility would be to start both searches in parallel. If the
> Gnutella search quickly returns a lot of (non-spam) results, discontinue
> the Kademlia search. If the Kademlia search quickly returns a lot of
> (non-spam) results, ask the UP to cancel the Gnutella search. Otherwise
> continue with both searches.
> 
> So there's actually no need to know in advance whether the item you're
> searching for is popular - you find out during the search.
> 
> The Kademlia search will find rare items, so we can probably reduce the
> range of the Gnutella search - it only needs to find "nearby" items
> ("nearby" currently doesn't mean much, just nearby in the Gnutella
> overlay, but in future could mean nearby in the social network if we
> connect to the same UPs as our friends).
> 


Hmm... 

We still need something for publish though, which is, believe it or
not, several times more frequent than search according to a Kad
measurement paper. Most likely we still need to figure out keywords'
popularity before publish or search. 

A node needs data to figure out keywords' popularity. If a node is a
leaf, then it relies on its Ultrapeers to collect data for it. For an
Ultrapeer, the data can come from 
1) its observation of keywords from its leaves and neighbor
Ultrapeers, as a benefit of the Query Routing Tables protocol. 
2) its observation of keywords queries passing through
3) ultrapeers gossip with each other about keyword popularity
The keywords and their popularity can be saved to and read from a file. 


Of course, there will be lots of work to design, to implement, and to
measure about the effectiveness and false positive rate, etc. Maybe a
feasibility study first? :-) 

Best,
Peng