Re: DHT usage and requirement lists

Quoting pwang708 <[email protected]> from ml.gnutella.dev-forum:
: Usage:
: 1) magnet links
: 2) push-proxies
: 3) alt-locs
: 4) searching for files by hash. 
: 5) Searching for file metadata by its hash. 
: 6) serving as distributed trackers for bittorrent enabled clients. A
: DHT can also be used to search torrent files.
: 7) helping Gnutella for rare keyword searches. 
: 8) serving as a decentralized contact-list so that one can search for
: specific users
: 9) searching for my friends' addresses or their Ultrapeers' addresses.
: Connecting to the same ultrapeers as my friends (and their friends)
: might allow me to find interesting files in fewer hops, which is good
: for two reasons: less bandwidth used, and fewer non-friends (potential
: spammers) reached by the query.
: 10) We can group peers having common interests and tag the groups. A
: group can publish the tags and a subset of current ultrapeers in the
: group into the DHT. A node can search the tags to join the group or to
: find files available under the tags.

I'd like to remove #6, #8, #9 and #10 for our first design round.

Keeping them in the back of your head is OK, but I don't want to discuss
them any further now.

Rationale: #1 (along with its subordinate #4 and #3) and #2 are the main
features we need to get right.  They come with their load of problems already
and spending brain time thinking about how we can efficiently achieve #10
or #6 will distract us from our goal and delay the practical first DHT
implementation significantly.

: Requirements:
: 1) Good user experience 
: 2) Low overhead.
: 3) Hotspot mitigation.
: 4) Write Quota system. A user can only publish a certain number of
: data items, to each DHT node.
: 5) Data security. E.g., only I can modify my push proxy information. 
: 6) Mitigation of "traditional" spams. I quote a line from Arne's
: previous post: "Spam, in the way it is used today, is a means to make
: it harder for people to reach some popular files for whose
: "protection" the spammers get paid." I think we should use some other
: name (such as targeted file poisoning/pollution attack) to distinguish
: it from "traditional" spams. 
: 7) Index poisoning/pollution attack mitigation.
: 8) Routing security. I.e., always be able to find the replica roots. 
: 9) A richer API. The DHT can be used for several things which may have
: different requirements. E.g., allow users to define their data's life
: time in [MIN, MAX], which are network wide parameters.
: 10) fixable lookup. For example, for lookups we don't care about
: speed, parallelism = 1, else parallelism = 3; exhaustive or not (find
: all replica root or find a node who answers a query); caching replica
: sets so that republishes consume less bandwidth... 
: 11) Identification system.

#10 is really DHT tuning.  When the time comes thinking about this, we can
easily change the client code, but it's not a requirement.

I don't think #11 is a requirement.  Identify for what?  Security?

#1 is very vague.  It's more a goal.

#2 is vague: low overhead on what?  Network?  CPU time?  Delay?  Storage?

I think #3, #4, #5 and #7 are the most important things.  If we get these
foundations right, the remaining will follow easily.

For #9, we're not trying to design a DHT where non-Gnutella nodes can plug in.
The API needs to be tailored to Gnutella's needs.  As for data lifetime,
I think each publisher should be able to specify a TTL, capped to 24 hours max.
And the STORE reply should indicate the TTL that was retained by the node
(could be adjusted down if the remote node thinks it's too far from the key).

Aiming for a 24-hour republishing by the original content owner is a guarantee
that stale data will not persist forever.  This implies that publishing
long-time static information (like file metadata, your use case #5 above)
is going to be impractical.  This is better handled by services like Bitzi.

I would like to add the following requirement:

#12) Typed data storage. A node must know the type of data it is storing and
what it represents.

This goes far beyond the "LIME", "TEXT", "TEST", "ANY", "BINARY" classification
done by Mojito.  This is not typing, this is more format specification.

I want to know that the information is an IP:port tuple.
And I also want to know that it is for a file, not a push-proxy.

That way, I can choose to not store torrent information at my node,
for instance.  Or to better balance storage space according to what I think
is important: alt-locs and push-proxies.

Raphael