Re: Re: Implementing a Gnutella-structured network only with basic HTTP possible?Date: Wed, 24 Feb 2010 09:08:15 +0000 (UTC)Date: Fri, 26 Feb 2010 12:55:15 +0000 (UTC)

> De : "Max" 

> in offload you connect to most possible nodes actively, but list all nodes
> known in the network. As chunks are stored 3x in the network, you have
> always a great chance to find the needed block in any node, you are
> connected to. so there is no isolated node. the isolated node is the offline
> one.

This does not work this way, because it's still impossible to predict or manage the "3x" metric.

What the P2P network is doing is trying to duplicate some chunk of data, but as there's no warranty that it will 
survive for long before the dissaminated chunks get lost (even caches will run in a MRU like way and chunks will 
discarded if they are not renewed), there must exist a way for the network to monitor this metric and maintain it 
under two minimum and maximum threshold (let's suppose that the network is fully connected, they should be reachable 
easily at anytime to detect their presence).

But this has a significant cost: just to make sure that some chunk of data is still present, you need a push or pull 
protocol to detect this. Traditionnally, Gnutella has used a pull strategy for disseminating data on request, so it 
lead to an explosion of pull requests (even if this is just a basic HTTP STAT request), but lacks a pull strategy 
which should be schedulable and should use multicast to a list of subscribers, and a way to unsubscribe.

Yes this is the role of a trcker to mange this, but trackers are not balanced, their problem is that they use the 
convential centralized server, instead of creating a mesh between participants (fully connected within a small hops 
distance) which will collaborate and schedule the broadcast of their presence information in that mesh.

The small meshes controling the presence of a chunk in the network should not be precisely located on dedicated 
machines. Its size must also remain small in order to minimize the traffic they generate internally for their own 
need (maintaining a chunk of data live). All of these participants must be also be equally reachable from anywhjere 
in the larger network. And if instructed by the original pusher of the chunk, a mesh should be able to "suicide".

These decentralized small sub-meshes should be able to elect any other nearby peer if one of the participants 
disconnects. When a peer will restart, it will also need to locate in the network where each of the meashes it was 
participating are located, and if it would be usefull to join them, or trade a superfluous chunk of data against 
another chunk from another sub-mesh to help them balance their workload (notably the lower and higher replication 
level of chunks should also follow the demand for some chunks, and the response time between participants in one 
sub-mesh should be monitored to see if the mesh is not about to explode and split into separate non-coordinated sub-
meshes that will use too much traffic on the network.

Do you need trackers in fact? No. The DHT is supposed to do that, and should be used not just to locate where the 
files are first injected, but also how they are split in chunks spread all over the network into caching meshes. The 
basic MRU strategy (where all chunk sub-meshes are considered equal) is not very good. The MRU strategy for the 
caches must take into account the persistance of life of meshes and their response time in case of increase of their 
workload (meaning degraded response time for the rest of the data belonging to less "popular" chunk sub-meshes).

Having a failure tolerant system effectively requires duplication (some will think about it as a solution named 
"backup" for long term storage, even if the location of these backups requires a longer response time), but this 
must not be done at the price of excessive duplication that will overwhelm the available ressources of the network.

A "3x" replication metric is not enough, but users of Gnutella typically consume more than what they produce. But if 
we estimate that we can increase the duplication, the limiting factor is still the bandwidth not the storage space 
which today is not expensive. But people are still reluctant to cache data for others, notably if there's a legal 
risk for this data. The only solution is to have this cached data completely split into small chunks whose content 
is not directlly trackable to the metadata describing it: the chunks have to be completely blind, possible even 
encrypted including during their transport on the network (the decryption keys are stored elsewhere as separate 
chunks, also unidentifiable directly with a backward link).

The only way to find and use any chunk of data (delocalized in decentralized and fully replicated sub-meshes, that 
can be located by the DHT) is to be able to describe any resource from a small staring metadata which will reference 
in only one direction the unique identifiers of subchunks containing the decryption keys, and the contents or 
additional metadata). And there asolutely MUST exist a feature in the design that will allow the ORIGINAL submitted 
to send a kill action (order all participants in all submeshes referenced by the initial ressource, to suicide 
themselves). Here also the DHT can help, and the kill action must be also secured.

Now there's the need to control the reputation of every participant: the public reputation is not good, what people 
want is a restricted level where their trust can be given to a handful of particopants, and where they can eliminate 
those that they don't want to participate in that trust. This is just like a structure where a user is a secured 
memeber of a group to which they can gight some rights, without necessarily extending the trust to further 
subgroups: the number of hops in the trust delegation tree must be controlable, as well as the possibility to browse 
the full list of those that have received some delegation. This job MUST not be performed by a central server, it 
should be only under the control of a participant.

The delegations also MUST not be symetric: you should be able to give trust for yuou own needs, to someone that will 
not trust you for his own needs. Trust delegations must be in a directed graph, non-transitive and be user-centric. 
You must be able to determine themselves and personnally who will be a secondary "administrator" of a group for 
which you must be always the primary administrator (from your own view of the group). The delegation system must 
absolutely not adopt the model used in Facebook "friends" (with infinitely recursive friends of friends via Facebook 
controling everyone). In that model, each user should be as powerful as Facebook, driving his own network and being 
alone to know exactly the full list of whom they have given trust to.

For this model to work, users must be able to socialize each other using one or more identities (not just one) and 
without necessarily revealing their true identity. These multiple virtual identities can be used to "test" some 
person, test some possible common interests, evaluate what they get in return, and be able to delete an identity or 
to join two identities into a single one. When joining identities, they should be able to give a rating for the 
networks of trusts they have in each identity, possibly in a non symetric way (for example join one secondary 
identity into a main one but not sharing with other users that know our main identity all waht we know from users 
that know our secondary identity, but allowing the reverse to be true, so that the secondary identity may share with 
its existing network of trusts some or all of the trusts we have rated in the prior primary identity).

In fact, a P2P network should really be used for that purpose. Then when you can trust people securely, you can 
socialize securely with them and exchange data through various ways, not just through Gnutella: you could provide to 
some of your friends specific rights, for example giving a tracker identifier (located vua the DHT) that describes 
an interesting ressource and allows the friend to download it from the meshes dissaminated in the network (and that 
cannot be retreived in a usable way without this initial tracker (which shoulod just consist in a secure digital 
print such as a TigerTree hash root, located via the DHT)

And in this network, you would no longer search for files, but for people that have some common interests on a 
topic. Users will then see on their application incoming questions from people they know and trust and looking for 
some ressources, they will reply by pointing them some lists of ressource trackers, or by pointing ressources that 
can be automatically shared between users within a group of contacts that the user has created for its group of 
trusts : instead of having to reply to all incoming questions, they can also give a limited (non-transitive) access 
to the existing list of ressources shared in that group.

When doing that, the trackers should not be usable as long as you don't belong to the group (thre should be a secure 
way to limit the number of hops, for example by providing decryption keys along each hop in order to successfully 
access to the content, and this can only work if the encryption/decryption keys are securely asymetric, such as RSA-
like key pairs, with a public key and a private key; the problem being that the time to generate a new key secure 
pair is quite expensive, so this should only occur when users add new contacts to a group or when they join two of 
their groups to create a posibly assymetric authorization between them along with a trust rating between each group)

Most of the traffic on the Gnet will then consist in exchanges of keys, and in encrypted discussions between 
participants in a group. In those discussions, people will be able to include and transmit trackers to specific 
people, along with the associated non-transitive decryption key, that the client will generate automatically).

Contact rating should also be more advenced than just "friends". We need several levels, as well as the need to 
delegate to some more reliable contacts the right and the numeric mean to subdelegate according to the limits we 
have given them and that they cannot ignore (because even them won't have the necessary key that will allow crating 
the first delegation): their contacts are not our contacts, our contacts are not theirs, and we should even be 
allowed to restrict communications coming indirectly via a trusted contact who had some limited level of delegation.

By design, the right of being forgiven completely will be warrantied (except for the readable content that will have 
been discussed between mutually trusted participants, because nothing will really warranty that they will have not 
save an archiving log of the past discussions, so if you reveal personnal details in those discussions instead of 
your virtual identity, all what can warranty the secret is a mutual agreement, or the laws about privacy, and the 
intelligence of people so that they won't exchange more details than what they receive from someone).

Finally, in that network, filesharing will just become an accessory service. Most people will in fact share blogs, 
opinions, their creation, demonstrate their skills, discuss politics, plan their holidays, criticize what they've 
seen on TV or a film seen at the cinema, they will socialize for meetings (and sexual plans of course), or to 
organize public events. Because they will know better each other, they will more securely share trustable files and 
their creations.

And when they are not connected (or connected with another identity on the network) they will become invisible and 
unreachable (outside possibly through a virtual mailbox stored for a limited time on the Gnet within the small sub-
meshes described above, with data or discussions that have a limited lifetime, say: min 1 week and max 1 month, 
according to data sender preferences, so that they can be automatically purged of all participating caches, unless 
the intended recipient loads it and securely kills immediately that data whose he is the only recipient)

Philippe.