Re: Re: Using the DHT to find download sources

pwang708 wrote:
> I saw many good ideas! Instead of going to the technique details
> directly, however, I'd like to take a step back and get requirements,
> etc clear, as I suggested in our email discussion. I think we can save
> time this way. 

You're right, I was only thinking about finding download sources but we
should work out how that fits into the bigger picture.

> Usage of the DHT:
> 1) helping Gnutella for keyword searches. A DHT can be used to publish
> and to search the rare files. We don't want the DHT to handle popular
> files because Gnutella is already good at searching popular files. And
> we don't want to overload DHT or create hotspots.

I'm inclined to agree with Arne here - Gnutella's pretty good at keyword
searches in the absence of spam, but it's difficult to see how we'd use
a DHT for keyword searches. Hotspots can't be alleviated by spreading
the postings for a popular keyword across several nodes (as we were
discussing with file sources) because you don't know which postings will
be relevant to a multi-word query until you've intersected them with the
postings for the other keywords.

For example if I'm searching for "debian lenny iso", it's not sufficient
to get all the postings for "debian", all the postings for "lenny" and a
random sampling of the many postings for "iso", because very few of the
postings for "iso" are likely to intersect with "debian" and "lenny".
Unfortunately I can't work out which ones until I've done the intersection.

On the other hand if I'm searching for a BitTorrent swarm or alt-locs
for a specific file, a random sampling is fine.

But maybe this problem was solved in the DHT literature years ago. :)

> 2) publishing and searching push-proxies, alt-locs, and magnet links. 
> 3) serving as distributed tracker for bittorrent enabled clients.

Yup, I can see how these could all benefit from using the DHT.
Interestingly they all involve storing node addresses under a key. Are
there any use cases where we'd want to store something other than a node
address?

> 3) Quota system.

I wonder what kind of identifiers we could use to enforce quotas. Using
IP addresses as identifiers might cause problems for NATed clients, but
if we use the IP:port or something generated by the client, such as the
GUID, then a client could use multiple identifiers to circumvent the
quota system.

> 4) Security related. 
>   i) Data security. E.g., One user's published data must not be
> deleted by another user; 

Good point. Do we need a delete operation at all?

>   ii) Index poisoning/pollution attack mitigation.

In my opinion this is another reason to use the DHT for download sources
but not for keywords: we probably can't prevent attackers from storing
bogus data in the DHT, so we need to make it pointless to do so, by
ensuring that anything retrieved from the DHT is easy to verify.

Keyword results are hard to verify: we can't tell whether a result is
spam until the user has downloaded (or at least previewed) the file. But
download sources are easy to verify, provided we know the TTH or
infohash: as soon as we detect an invalid block we can blacklist the
node that provided it and ask the DHT for more sources if necessary.

Of course that doesn't answer the question of how we get the TTH or
infohash in the first place - we'll need to do a keyword search at some
point. This is where buddy lists, social clustering and reputation
systems come in... but let's save that for another thread. ;)

>   iii) Routing security. I.e., always be able to find the replica roots.

Another big problem. :) Have you come across any secure DHT designs that
don't require a certificate authority?

Cheers,
Michael