Re: Using the DHT to find download sources

--- In [email protected], Michael Rogers <m.rogers@...> wrote:
>
> pwang708 wrote:
> > I saw many good ideas! Instead of going to the technique details
> > directly, however, I'd like to take a step back and get requirements,
> > etc clear, as I suggested in our email discussion. I think we can save
> > time this way. 
> 
> You're right, I was only thinking about finding download sources but we
> should work out how that fits into the bigger picture.
> 
> > Usage of the DHT:
> > 1) helping Gnutella for keyword searches. A DHT can be used to publish
> > and to search the rare files. We don't want the DHT to handle popular
> > files because Gnutella is already good at searching popular files. And
> > we don't want to overload DHT or create hotspots.
> 
> I'm inclined to agree with Arne here - Gnutella's pretty good at keyword
> searches in the absence of spam, but it's difficult to see how we'd use
> a DHT for keyword searches. Hotspots can't be alleviated by spreading
> the postings for a popular keyword across several nodes (as we were
> discussing with file sources) because you don't know which postings will
> be relevant to a multi-word query until you've intersected them with the
> postings for the other keywords.
> 
> For example if I'm searching for "debian lenny iso", it's not sufficient
> to get all the postings for "debian", all the postings for "lenny" and a
> random sampling of the many postings for "iso", because very few of the
> postings for "iso" are likely to intersect with "debian" and "lenny".
> Unfortunately I can't work out which ones until I've done the
intersection.
> 
> On the other hand if I'm searching for a BitTorrent swarm or alt-locs
> for a specific file, a random sampling is fine.
> 
> But maybe this problem was solved in the DHT literature years ago. :)

For example, say the published DHT value includes the full file name,
then you search the DHT with key=sha1(debian), when you find the
replica roots, send a findValue including the list of keywords, in
this case "debian", "lenny", and "iso". The replica roots search their
database and find the DHT value with filename="debian lenny iso"
matching all the keywords, then return the value. Kad does something
similar. Arne posted a link which explains a lot about Kad. 

> 
> > 2) publishing and searching push-proxies, alt-locs, and magnet links. 
> > 3) serving as distributed tracker for bittorrent enabled clients.
> 
> Yup, I can see how these could all benefit from using the DHT.
> Interestingly they all involve storing node addresses under a key. Are
> there any use cases where we'd want to store something other than a node
> address?
> 

I once had my computer crashed, that made me think about something
like a "remote playlist", where I store my file list in DHT. Only the
metadata are stored. Then when I get a new machine, I can at least get
my file list back, and then client can tell me what files are still
available in the network. The list can also be used if I am away from
my home. A toy idea... :-)

> > 3) Quota system.
> 
> I wonder what kind of identifiers we could use to enforce quotas. Using
> IP addresses as identifiers might cause problems for NATed clients, but
> if we use the IP:port or something generated by the client, such as the
> GUID, then a client could use multiple identifiers to circumvent the
> quota system.
> 

This is always a problem. 

> > 4) Security related. 
> >   i) Data security. E.g., One user's published data must not be
> > deleted by another user; 
> 
> Good point. Do we need a delete operation at all?
> 

Not really a important operation, since DHT values expire. But it
could be useful if we put a quota on each client. I think I should
have used the word "modify" :-) E.g., only I can modify my push proxy
information published in DHT...  

> >   ii) Index poisoning/pollution attack mitigation.
> 
> In my opinion this is another reason to use the DHT for download sources
> but not for keywords: we probably can't prevent attackers from storing
> bogus data in the DHT, so we need to make it pointless to do so, by
> ensuring that anything retrieved from the DHT is easy to verify.
> 
> Keyword results are hard to verify: we can't tell whether a result is
> spam until the user has downloaded (or at least previewed) the file. But
> download sources are easy to verify, provided we know the TTH or
> infohash: as soon as we detect an invalid block we can blacklist the
> node that provided it and ask the DHT for more sources if necessary.
> 
> Of course that doesn't answer the question of how we get the TTH or
> infohash in the first place - we'll need to do a keyword search at some
> point. This is where buddy lists, social clustering and reputation
> systems come in... but let's save that for another thread. ;)
> 

I agree. It is hard to defense. But I hate to cut off features. :-(

BTW what happened to Credence? 

> >   iii) Routing security. I.e., always be able to find the replica
roots.
> 
> Another big problem. :) Have you come across any secure DHT designs that
> don't require a certificate authority?

Unfortunately no. :-( I think some social network based solutions are
promising. I doubt if they are practical. 


Best,
Peng