Re: Re: Using the DHT to find download sources

[email protected] wrote:
> ... that would make the nature of the data stored opaque.  I prefer a
> scheme where the nature of the data stored in the DHT can be monitored
> and analyzed.  For instance, we can enforce different architectural TTL
> limits for different types of data.
> 
> Knowing the type of data stored can also allow validation of the data
> at store time, rejecting badly-formed values.  When storing an alt-loc
> for a file for instance, or a push-proxy address, we can ensure the
> IP is not that of a private LAN (which would not be reachable from the
> outside).  If you don't know the data you're storing, this kind of sanity
> check is impossible.
> 
> One problem with validation is that it can prevent extensibility.  We
> cannot foresee today all the possible value types we're going to be
> willing to store in the DHT, so it is not possible to validate types
> we do not know about.  However, if we choose the route of this "explicit
> typing" of values, we have to be clear that an unknown type is something
> that is totally acceptable by DHT nodes, only it won't be validated and
> therefore "semantic errors" will not be detected as early as they could.

All good points. Personally I prefer end-to-end designs, but it's really
just a matter of taste; I'm happy for the metadata to be explicit.

Should we add a DHTValueType to the FIND_NODE_REQUEST?

> If one follows the
> pure logic behind the existence of the token, its lifetime should be
> short, i.e. a few minutes.  That would physically prevent someone to
> store on a node without going through the FIND_NODE protocol first
> (which is useful in my little DSHT design here because "hiding" of full
> hosts can be done during FIND_NODE, precisely).
> 
> Such a decision would make caching Security Tokens totally useless.

At present I believe the SecurityToken is calculated by encrypting the
client's IP:port with a secret key. This allows the DHT node to verify a
token statelessly by decrypting it with the secret key and comparing the
result to the client's IP:port.

If we want to enforce a particular lifetime for tokens (or make them
single-use), DHT nodes will have to keep track of which tokens are
valid, which just shifts the burden of caching from the client to the
DHT node.

We could, however, recommend that clients don't try to cache or reuse
security tokens - they should always send a FIND_NODE_REQUEST before a
STORE_REQUEST and use the returned token. That leaves DHT nodes free to
use any method of generating and checking tokens.

> There is already an architected
> "Contact's flag" field in the header of all Kademlia messages in the
> LW documentation.  However, I am not sure we can steal one bit here to
> report "busy" conditions if these pertain to only a specific key and not
> to a general flag about the node.

Since we're making other changes to the protocol anyway, let's do the
right thing and add a new flag.

> We need to extend the set of error codes that can be
> returned.  I'm not satisfied by having to report "0x2 - Error" on STORE
> if that can mean several conditions, including "Full" for the key, or
> "Full" as in: no more storage available for any key.

OK, how about 0x03 KEY_FULL and 0x04 NODE_FULL?

> We need to distinguish FIND_NODE for bucket refreshes from FIND_NODE
> preparing a STORE operation.  Because we don't necessarily want to hide a
> node that is the hotspot for a given key in bucket refreshes, or the node
> that precede it in the KUID space will soon become the hotspot as everyone
> will start to think it is the ideal place to store and lookup for the key!

I'm not comfortable with the idea of hiding nodes, because the client
might discover them by other means (eg it might have other lookups in
progress). In my opinion we should return the best addresses we know
about, but if we set the "full" flag the client should assume that the
returned addresses are also full and shouldn't contact them during this
lookup (although it can still add them to its routing buckete for future
lookups). At the end of the lookup it should store the value at the k
best non-full nodes.

Cheers,
Michael