Re: Using the DHT to find download sources

--- In [email protected], Arne Babenhauserheide <arne_bab@...>
wrote:
>
> Am Montag 21 Juli 2008 18:49:12 schrieb pwang708:
> > Usage of the DHT:
> > 1) helping Gnutella for keyword searches. A DHT can be used to publish
> > and to search the rare files. We don't want the DHT to handle popular
> > files because Gnutella is already good at searching popular files. And
> > we don't want to overload DHT or create hotspots.
> 
> Wouldn't implementing a decentral spam detection system ( like 
> http://credence-p2p.org ) solve the rare searches problem far more 
> efficiently? 
> 
> With that, spam wouldn't be counted as results anymore and dynamic
querying 
> would no longer stop because of spam results. 
> And that would mean, QRP would again avoid hitting users who can't
have the 
> file, and searching for rare fiels would work again. 
> 
> Spam is the only real problem for keyword searches in Gnutella. 
> 
> And from my experience when Credence was still active, it did a
quite good job 
> at detecting spam. 
> 

Good point! But I am not really sure if reputation based anti spam
systems work well for rare files. Any ideas or links?

> 
> Gnutella is great at keyword searching, more exactly: keyword
searching is its 
> main function, and it is heavily optimized for that. 
> 
> Can the DHT really beat Gnutella when it comes to keywords queries? 
> 
> Before you answer, please consider Gnutellas bandwidth usage: 
> - Ultrapeer: about 7kB/s if you add up and down
> - Leaf: max. about 1kB/s if you add up and down. 
> 
> (these are numbers I observed myself - not really scientifically
measured, 
> though - not enough test cases, but with a big share of free files)
> 
> - source: http://gnufu.net/Why_Gnutella_scales_quite_well
> (besides: I just reworked that a bit)
> 
> And if it can't: why should we use two tools for the same task? 
> 

A DHT can find rare files efficiently. A DHT search takes several
hops, say 3 or 4. Then in Gnutella, unless a copy of the file are
within less number of hops from your machine, you will need more time
to find it. In addition, if a rare file exists, then a DHT is able to
find it. With Gnutella, it is expensive to provide such guarantee.
Note that we are only talking about rare files. Gnutella is very good
at finding popular files.

> 
> > 2) publishing and searching push-proxies, alt-locs, and magnet links.
> > 3) serving as distributed tracker for bittorrent enabled clients. A
> > DHT can also be used to search torrent files.
> 
> 4) Searching for files by hash. This might be implicitely part of 1)
or 2) , 
> but I think it's important enough to deserve to be mentioned
explicitely. 
> 
> 5) Searching for specific users (which could be used in decentral
contact-list 
> functions, for example instant messaging). 
> 

Good idea! 

> > Requirements:
> > 1) Fast search and low overhead.
> > 2) Hotspot mitigation.
> > 3) Quota system.
> 
> What exactly do you mean by that? 
> Allowing only a certain number of searches? 
> 
> Or rather something like avoiding overloading clients? 
> 

Write quota. A user can only publish a certain number of data items,
to each DHT node. Yes, avoiding overloading clients...

> > 5) Spam mitigation. This is also security related.
> > 6) A richer API. The DHT can be used for several things which may have
> > different requirements.
> 
> 
> Best wishes, 
> Arne
> 
> -- Weblog: http://blog.draketo.de
> -- Infinite Hands: http://infinite-hands.draketo.de - singing a part
of the 
> history of free software. 
> -- Ein Würfel System: http://1w6.org - einfach saubere
(Rollenspiel-) Regeln
> 
> -- PGP/GnuPG: http://draketo.de/inhalt/ich/pubkey.txt
> 
> 
> [Non-text portions of this message have been removed]
>