Re: DHT expiration questions

> 
> Since a DHT Value does not carry the original creation time of that
> particular value, how do LimeWire nodes decide what to expire and when?

For every value stored, a node assigns it a creationTime (=
System.currentTimeMillis()). So Mojito DHT values have creation time
but they are the replica roots' local time. 

To decide expire time, a node tries to find k nodes closet to the key
(of value) in it routing table. If it found less than k Nodes or the
local Node is member of the k-closest Nodes then use the default
expiration time (one hour). Otherwise, the value expires inversely
proportional to the xor distance. More detail can be found in
getExpirationTime() in DatabaseUtils.java. 

> 
> My fear is that with all the replication that goes on between the
> k-closest
> nodes, it is easy to have stale data being permanently carried by
the DHT,
> preventing new fresh data from being stored (all the keys being full).

A stale data item should not be permanently carried, unless some node
keeps republishing it. Then it is not stale. :-) 

It will be a different story if we have attackers in the network. 

> 
> I start to notice many STORE requests at my node being refused
because the
> keys are full, precisely, and this particular node has been up for less
> than a day...
> 

I saw some ERROR in store response too. E.g. assuming the number of
values can be stored under a key is N, when more than N nodes try to
publish as Alt-Locs, they get ERROR.   


> Also gtk-gnutella is validating that the creator's contact address is
> valid
> and is refusing STORE requests coming from 127.0.0.1, for instance, with
> STORE_SC_BAD_CREATOR. I understand this could be because the node is
> firewalled, but given that DHT PONGS carry the requestor's external
> address,
> firewalled nodes could at least use that for their contact address.
> 
> I see a lot of lines such as:
> 
> DHT traffic from new firewalled node
> cde668edac5f8391f0ac55af9b34270f8191db28 at
> 127.0.0.1:0 (LIME v0.0)
> 
> in my logs, and this shows that these nodes do not learn their address
> correctly.  Which is a problem. GTKG for now will remedy to this
> situation by patching the contact address on the fly to be that of the
> UDP datagram source.

That is good. 

> 
> Anyway, what is the policy LimeWire is using to expire data?  What are
> the republishing and replication parameters?  How do you prevent stale
> data from being replicated over and over (e.g. if my node comes up and
> is given STORE from its k-closest, how can it know if that data is about
> to expire at the original node?).
> 

The default republish time is 30 minutes. The republish time could be
smaller if the DHTvalue was published to less than 20 node last time. 

(locationCount * DatabaseSettings.VALUE_REPUBLISH_INTERVAL.getValue())
                    / KademliaSettings.REPLICATION_PARAMETER.getValue());


As far as I understand, only the root node (whose ID is the closest to
the key) forwards data to new replica roots (whose ID is among the 20
closet ones). So a new node joining the neighborhood becomes the root,
then the data will expire later what one hour. So there is a chance
that data being replicated over and over. 

> P.S: I've seen STORE requests for BINARY data with a length of zero
bytes?
> What is the purpose of storing empty BINARY data? Is that a bug?

Interesting. Maybe a bug. I will take a look at data I collected. 
BTW, when I was studying another network. I saw many things that
didn't make sense. Later I found that they were sent from a research
lab which was studing the network at the same time.