Re: DHT expiration questions

Quoting pwang708 <[email protected]> from ml.gnutella.dev-forum:
:The figure is based on these data. The green line shows the STORE
:requests my node would received if we used different expiration time.
:Assuming we need 100% STORE requests when we use 30 minutes as the
:expiration time, we would saved about 5% of STORE requests when we use
:1 hour as expiration time. Using 2 hours expiration time, we could
:saved 7%. After that we cannot save more, since most nodes stop
:republishing, and most likely left the network. 

I haven't looked at the graph yet but I have three comments:

* We need to differentiate the expiration time by the type of data.
  An ALOC should be expected to be valid longer than a PROX, for instance.

* If the node issuing the STORE was indicating the desired TTL for the
  value, we wouldn't have to make bad guesses that will prove wrong most
  often than not.

* LimeWire nodes seem to have a fauly node lookup algorithm.  I'm seeing
  tons of original STORE (not replications but publishing by the creator)
  for IDs that are way outside my k-ball.  For instance, my node ID's
  starts with 35... and I just got a STORE for f0....  Not even 1 leading
  bit in common!  Data was published at 22:09:15 and expired at 22:11:10
  due to the exponential decimation algorithm I described in another
  message.

Until that last point has been diagnosed correctly and is fixed in the
network, we cannot base our reasonning by analysising what's happening
in the current network since there seems to be a critical problem in the
DHT foundations, in LW's implementation!

A proper node lookup algorithm implementation is a requirement for proper
DHT operations.

Here are some data for my GTKG node: I've been up for 6 hours and I hold
2033 values spread over 1128 keys.  I have received 11529 replications for
the data held, and 4881 republishes.

Given that I count a republish only when the data is still there, not
expired, and sent by the same creator, and knowing that I am expiring
keys outside my k-ball quickly, that's a LOT of republishing over 6 hours.
(I have more ALOC than PROX: I received 3042 STORE for PROX, and 43619 STORE
for ALOC).

But given that LW appears to have a fauly node lookup algorithm, it's
hard to draw a conclusion.

The problem I have with fast republish is that I have a node sharing many
files, that my node has an average uptime of 24 days, and that I don't want
to republish all the files every 30 minutes because of bandwidth concerns,
yet at the same time I want my files to be visible. A STORE operation is
costly, because a node lookup is typically using 4 KiB out and 20 KiB in,
then STOREs are at least 150 bytes (depends on the value length, really)
and you have to issue 20 of them, so that's 3 KiB out and about 2 KiB in
for the acks.

The net publish cost for a file is therefore 7 KiB out and 22 KiB in.
Multiply that by 1000, and to publish every 30 minutes costs you an
average 3.88 KiB/s out and 12.22 KiB/s in.  I think it's too large a cost.

It is a huge design mistake to not have included intended TTL information
in the values being published...

Raphael