Re: Implementing a Gnutella-structured network only with basic HTTP possible?Date: Wed, 24 Feb 2010 09:08:15 +0000 (UTC)Date: Fri, 26 Feb 2010 12:55:15 +0000 (UTC)

Quoting Michael Rogers <[email protected]> from ml.gnutella.dev-forum:
:Hi Arne, Raphael,
:I think you both have excellent points. The complexity of the protocol
:is too high, and it does discourage new developers and new ways of using
:Gnutella - but the wire protocol's only a small part of that complexity.
:The larger problem is that a peer must implement several rather
:complicated features before other peers will even talk to it. There's no
:equivalent in Gnutella of a dumb router that just forwards packets
:without knowing about the higher-layer protocols those packets
:implement, or a dumb client that uses higher-layer protocols without
:knowing how they're implemented at the packet level. Layering is
:completely broken, so it's all or nothing - and "all" is too much for
:most developers to chew.

You're also bringing up an excellent point that I did not include in
my original analysis: the Gnutella protocol requires sophisticated
application-level knowledge to be maintained and used, and is not layered
as cleanly as a network stack would typically be.  Therefore, implementing
the protocol itself requires a full end-to-end understanding.

It does not mean that the application which is implementing the protocol
cannot be layered (hopefully, otherwise it would be totally un-maintainable),
but that decisions to process a packet require information available at a
higher level, thereby spreading the complexity instead of encapsulating it
in a well-defined autonomous layer.

Those of us who were lucky to start working on Gnutella in the early days
were given the time to gradually understand the features as they were
introduced.  I can imagine how complex it would be today for someone to
implement Gnutella from scratch.

:So we seem to have a rather unappealing choice. We can let the network
:stagnate, as fewer and fewer new developers get involved. We can abandon
:the network and start from scratch - but given the huge number of people
:still using Gnutella and the uncertainty of persuading them to move to
:another network, that seems like a desparate measure. Or we can try to
:plan a gradual transition from the current network to a simpler, more
:maintainable one, with lower barriers to entry.

Over the past two years, I've been implementing the DHT layer in gtk-gnutella
(I'm not working full-time on this, only during my "leisure time", with
activity peaks here and there) but I would say I roughly spent the equivalent
of 8 man.months.  That's the "garage development model": I'm working alone,
and I'm responsibile for keeping track of the development progress and the
order in which I implement things.  I imagine that in a corporate structure
with teamwork and (necessary) organizational overhead, the figure would be
easily tripled, if not quadrupled.

What's interesting is that the DHT is based on a standard design (Kademlia)
with (thank you LimeWire!) a farily correct description of the binary message
format and a high-level description of special non-Kademlia features (like
the security token).  It has a well-defined API, and the Gnutella usage pattern
is well defined (well, that took me some time because I had to reverse-engineer
the binary packets -- LimeWire had not documented that part at all).

The point is: despite the fairly well-defined boundary of the layer and the
well-understood Kademlia design, the actual implementation and tuning of the
code required a fair amount of effort.  The good news is that it could be
eventually turned into a library.  The bad news is that doing that would
still require some effort, cleaning the APIs and, most importantly, adding
a well-defined configuration API (since today it uses the gtk-gnutella
configuration layer directly).

Can Gnutella be "redesigned" so that it can be made of more independent
components, with a well-defined interface (from an architecture standpoint)
to other components?  Can it be done in a fairly "backward compatible"
way? (probably not, since then you fall again into the all-or-nothing pitfall).

Perhaps one way to go would be to invent Gnutella 3.0 (I believe 2 has been
taken) and make sure we produce "hybrid" servents that can continue to use
the Gnutella network protocol but can also talk the new protocol.  As long
as we keep the standard HTTP interface we've been using for exchanging files,
the Gnutella 1.0 and Gnutella 3.0 would be able to freely exchange files,
thereby not alienating users too much...

Now who would "lead" the Gnutella 3.0 effort?  This must not be an "I implement,
you follow" approach.  We saw where it led with the avorted "2.0" mutiny.

Would that effort bear reasonable chances of success that would justify the
necessary investments?

Raphael