re: Re: Implementing a Gnutella-structured network only with basic HTTP possible?Date: Wed, 24 Feb 2010 09:08:15 +0000 (UTC)Date: Fri, 26 Feb 2010 12:55:15 +0000 (UTC)

This discussion is going nowhere. The proposer is not in fact proposing anything concrete, and has a very poor 
understanding of what is a P2P network and why it works or does not work.

In fact he must realize that the exact binary protocol format itself does not matter at all. The fact that messages 
are formatted in one way or the other is absolutely not important. The difficulties are elsewhere.
Gnutella in fact has capitalized experience in areas that requires deep mathematical and statistic analysis to help 
define and refine a policy so that the network itself does not explode.

May be he never had experience of how the early Gnutella evolved: it initially started with a pure P2P network where 
all servents where completely equally powerful and could leech as much as they could, and it rapidly exploded: the 
protocol could not scale and could not adapt to the fact that users are connected with very different capabilities 
and had a very different usage pattern, with a few sharing a lot, but also donating extremely high level of 
bandwidth with little or no return in terms of value.

Things have been learnt in Gnutella by studying various models, but all the P2P protocols have concluded that 
effectively the HTTP protocol was a fundamental root for actually transmitting the data for file transfers or other 
services.

But the difficulty is definitely not in the file transfer itself. for that we have a very strong protocol, described 
by a RFC which is extremely precise and has capitalized decenials of active researches from many companies and 
searchers around the world, but also from worldwide efforts made in the Internet infrastructure, gateways, routers, 
modems and access routers, proxies, in order to maximize its efficiency. The HTTP protocol (in its 1.1 version, 
excluding the early 1.0 version which had problems) is then very strong, but it absolutely does not solve the 
problem of P2P which is to transform the hierarchical client-server model (more or less working in a star topology) 
into a servent model (more or less using a semi-random mesh topology).

HTTP is so successful that it has now nearly replaced FTP as the prefered file transfer method (FTP still has some 
minor advantages because it offers a way to browse lists of ressources, something the HTTP still does not offer with 
its basic GET/PUT/HEAD/POST methods; however lists of ressoruces are now handled by other standardized protocols 
like HTML or XML schemas, where resources are represented as links containing an URL: you can use HTTP to retrieve a 
list of ressources within a single ressource that has to be parsed, all you need is a data schema for these lists). 
With a standard schema (for example UPnP), HTTP is now used as a networking filesystem (and it competes now with 
CIFS/Samba for Windows-based networks as well as NFS in the *nix/Linux world, and it could be their successor with 
the new success of NAS devices deployed along with wireless terminals).

P2P protocols are still an active research (even Microsoft has developed its own protocol to try filling the gap not 
filled by HTTP). The huge problem in a P2P network is definitely not the file stransfer itself: you can do it very 
efficiently as long as you a locator for the ressources you want. The problem is elsewhere:
* how to connect to a destructured mesh if all servents are equal ? There's no liability of anyone if a HTTP server 
is not responding, the Internet infrastructure does not provide any ressource to help provide alternate entry 
points.
* how to keep a mesh fully connectd without creating many isolates ? You can think about it as you want but the 
problem is always the same and always involves the same solution: you need to allow the network to self structure in 
order to have some nodes layered that will service others by capitalizing on their experience, and keeping caches of 
what they know about the network, and trying to reconnect the isolates.
* Creating a layered structure within the mesh is necessary to compensate for different capabilities (notably their 
bandwidth, but also the number of connections they can support) and for different uptimes of servents.
* In a mesh topology where you never know exactly where you'll find a file or service, locating the node(s) capable 
of responding that file/service requires some duplication of messages to try alternate paths. This duplication is 
the most difficult problem to solve, and this is where you'll very complex maths, with statistics and usage 
policies. The protocol, independantly of how it is coded, MUST include minimum and maximum response times and must 
absolutely balance the usage made by all servents (including itself for its own use) 

HTTP is based on TCP, which requires a dedicated port for each session. TCP has a slow setup time and very slow 
disconnection time. Each time you open a session, its associated port (on the client when initiating its session) 
reamins associated with a unique server for a very long time. The number of ports per host interface is limited on 
each client and this becomes even worse when clients are connected through NAT routers, because these routers are 
even more tightly limited: some of them will limit the usable number of client ports per client to any a handful. 
This means that a TCP client cannot "speak" with lots of servers at the same time, and that their "outgoing degree" 
is very limited, even if a TCP server can accept lots of simultaneous clients on the same incoming port number.

TCP is then very unbalanced. The way to compensate for it for P2P meshed topologies is to bypass it completely, but 
this also causes lots of problems because the only viable alternative is to use sessionless protocols: UDP is fine 
there (we could use other datagram-based protocols, but there's no incentive to do that as it would require raw 
datagrams, and the Internet infrastructure and the access routers are not designed to support it); ICMP could be 
used as well, and Gnutella still does not use it, simply because it is blocked on too many routers and firewalls and 
there's no alternative (ICMP has not been designed as a transport layer but as an internal brick to help debug some 
part of the routing paths).

Other interesting protocols are IGMP (for efficient UDP multicasting): Gnutella still does not use it where it 
could, except for host discovery in some Gnutella servents. But anyway, given that the goal of Gnutella is 
definitely not the file/data transfer service, but the discovery of hosts, the maintenance of the mesh to keep it 
well interconnected, and then a way to perform networkwide searches efficiently without exploding the whole network 
with too much duplication of messages.

Algorithms have been developed to help minimize the duplication of messages, by trying to "guess" what are the best 
paths in a mesh. In fact Gnutella was not the first P2P network written to perform this type of guess. In fact the 
Internet itself is born and is still itself natively a P2P network. Think about how an IP address can be routed 
between nodes within a mesh of links, and you'll realize that the P2P network is already in the IP routing discovery 
mechanism: we have not enough learnt from it, and still not realized that the internet works betcause of several 
competing P2P networks trying to do the same thing, the gateway-to-gateway protocols, and notably the binary 
protocol used to advertize and convey IP routing discoveries from network to network, and recognized by routers.

Internet itself would have exploded only because of the traffic generated for IP routing discovery and maintenance. 
Yo solve the problem (which is still very acute, and will persist even with IPv6), it was necessary to layer the 
network and give liabilities to some more capable nodes that can support higher traffic and act as interchanges for 
routing informations. Other protocols have come to the help, because the P2P network of Internet could be easily 
exploited to perform malicious attacks, and notably another fundamental brick: the DNS system, which is, in my 
opinion, the best protocol ever which is extremely scalable.

Gnutella has still not evolved to integrate the benefits of the DNS system and notably how it can cache informations 
and delegate them to others while also maintain it secure against attacks.

It has already solved the problem caused by TCP unbalanced topology, by allowing the use of UDP to solve the problem 
of limited port numbers. But TCP did not completely get rid off, it was in fact rewritten in its lower layer to use 
UDP within an encapsulation. And onc this was done, it became possible to use that encapsulation as well for 
transporting everything we already transported in native TCP.

Another problem to solve was how to extend the reachability. Given that the number of NAT-routed clients have 
exploded, if we had left this in that state, nobody could connect now, because almost nobody would find a reachable 
target whose ports are already all taken and that can't service more people. Thanks the Internet invented UPnP, so 
that we can now work with and manage NAT routers and firewalls without asking people to manage manually their port 
forwarding rules (and possibly break by error their security...)

Things on which we must still work are there:
* integrating the DNS system (or a similar system) to make the network realiable and more structured
* integrating IGMP for efficient broadcasting (mking use of UDP multicast)
These will make the network more resistant to attacks (and to leechers)

But things must still be worked on. Currently Gnutella is still used to find files that can be easily identified and 
copied. This allows locators (URL-like) to work for file discovery. We must find a way to help disvering services. 
Notably to help locating people instead of just files. Services or people are described by a sort of business card 
that can be thought as a sollection of metadata containing all the contact addresses, and which is perfectly 
replicatable as if it was a file. With such an extension, we could build a decentralized social network, and avoid 
the (dangerous) centralized approach currently followed by Facebook.

We must remember why do we need a P2P network: it's to protect our freedom against giant ultrapowerful networks that 
have exclusive knowledge of who we are and what we are supposed to see and want to see on the Internet.

Killer apps for Gnutella or any open P2P network would be to develop an open alternative to Google, Facebook, VoIP 
operators (they are now the same as traditional telcos...), and CDNs (for broadcasted radios and TVs, or for massive 
delivery of softwares or automated software updates, but here there are now near cost-free solutions for hosting 
them on webservers whose storage space is now virtually illimited). Our danger is for our freedom: can you trust a 
single private network that knows much more things than your own governement, but makes what he likes with your 
personal data and becomes a single point of dependance? What will happen and how will you live with essential 
services if you suddenly loss your presence on this single point of presence?

(For example we are now told by job seekers to open an account on Facebook... I really don't like that, and I want 
the right to forgive my past, I want to be able to change my opinion, change my hobbies, I want to remin open to 
newer or older interesting things, I want to reamin unique and to be a mistery for those that will meet me; I don't 
want to know everything about those I will meet, and if later I'm no longer pleased to live or work with someone, I 
want to be able to cut the links completely without carrying a huge weight behind me and a past that I will prefer 
to have forgotten when it no longer matches my new desires. We all change through our life, but archived data on a 
huge single central server that would know everybody and where anyone can know everything and judge you only by your 
past or by a few minor aspect taken randomly from it out of its context, is something really frightening: this is 
Orwell's world).

Don't think that a single protocol will solve everything. In fact it is completely normal that Gnutella does not 
work alone and that it depends on lots of other specifications, because the Internet itself is also not working as a 
single protocol, and because it is highly heterogeneous with nodes that definitely have very differnt capabilities 
and bandwidth.

Restarting a new protocol from scratch will solve nothing, in fact it will just refragment the network and will 
cause new interoperability problems. If the Internet has succeeded, its not because of the newer protocols that have 
been progressively added along time, but because it evolved gradually by ALWAYS offering a good interoperability 
with the existing assets: new protocols have been sometimes been tested and then either massively adopted or benn 
progressively abandonned. The same is currently happening with the slow start of IPv6.

Thanks for this (long) reading of my opinions.

Philippe.