RE: Re: streaming over Gnutella

[email protected] wrote:
> Envoyé : lundi 8 octobre 2007 09:29
> À : [email protected]
> Objet : [the_gdf] Re: streaming over Gnutella
> 
> Quoting Arne Babenhauserheide <[email protected]> from ml.gnutella.dev-
> forum:
> :Are there any other problems with it?
> 
> Yes: since query by URN has been shutdown by LimeWire and does not work
> well
> anyway, we need to figure out which filename(s) is/are going to be used
> to index the partial files shared. (think magnet, where you don't
> necessarily
> know the file name).

Query by URN using Gnutella keyword searches has been replaced by query over
a overlay network, which works now.
You can argue that the overlay network is not Gnutella, but LimeWire has
demonstrated that the overlay really helped making Gnutella more robust.
I think that the association of Gnutella and the overlay (named Mojito in
LimeWire) is the best advance that makes such thing possible now.

And may be it's time to integrate more tightly the two, and defining a
policy about when classic Gnutella searches fits the best, andsearchs
through the overlay network is the best.

Anyway, my proposal did not include ONLY a need for searching by URN. In
fact, before streaming can happen, where streams can be distributed and
sources of the streams can be located (using the overlay network), users
will still need a way to discover these streams, when they are not initially
identified by a URN.

And for such thing, we still need classical Gnutella searches by keyword:
streams are, like other files, discoverable by their name, description, and
other metadata. So we'll still look for streams matching keywords and other
filters based on non unique characteristic. Then users will get in their
results a list of files or streams matching their searches.

The difference will be that streams don't have any defined file length. If
you attempt to download from them you'll get 0 bytes, because what will be
part of the query hit will be:
* the name of the stream (like files)
* its file size : 0 byte
* meta data
* and a service URN (similar to the service id in a numeric TV multiplex).

For downloading the stream, the name will have no value: if you connect to
the source and request it, you'll get 0 bytes in return (but you'll get
confirmation that it is associated to the service URN.

It's important to see that a service URN does not indicate directly the
source of the data, or even one of its multiple candidate encodings. There
may exist several codecs used to generate the same service. Each origin
codec is a participant to the service. A service may be composed of several
sources generating parallel streams that are considered "equivalent" despite
their content are distinct. Such thing could be used to allow the network to
support the same event with different encoding qualities enoded at different
bitrates: to be usable indistinctly, these sources must still agree with a
single epoch reference for timestamps.

Time should bedefined in all streams (whatever they are, and even if they
are for different service URNs) using a common unit: I propose the second.

Each service must also propose a single frame rate. Typically, the frame
rate of a video stream will be 25 frames per second, independently of the
encoding bitrate, and audio stream would be typically 75 frames per second,
also independently of the bitrate. The frame rate will constitute a
constraint for making multiple origin sources compatible with each other so
that they can participate to the same service URN.

The difficulty is still not there: the most important difficulty is about
defining a single time reference. That's why, in a first approach, a service
URN could support only a single origin source (i.e. a single codec). But
suchthing is not impossible to guarantee, notably when the same host is
effectively encoding the same live event into multiple streams at varying
rates.

So when you have a service URN from a peer's query hit result, you can
connect to that peer to download a stream fragment from this URN (not from
the name of the stream). You'll get a reply containing the first fragment,
but also the URN associated to a source codec, and an URN for its associated
TigerTree data.

Note that the TigerTree data is another stream: it must use the same
timereference as the main stream you are downloading.

You can use the overlay network (not Gnutella) to look only for the
locations of a service URN: when you'll connect to each of these locations,
you'll be able to download fragments for that service, however nothing will
indicate that they are from the same source, just that they participate to
the same service, so you may download in fact fragments from one of the
separate parallel streams that make it and made by its multiple
participants, so there must be some information to allow you select which of
the multiple parallel streams you want.

Such information is metadata about the origin source's codec, including the
service URN itself, however the service URN must be the same for all origin
sources.

You also don't know if the discovered locations is a origin source or not,
you'll discover only locations that are part of a download mesh where the
origin source is present, indistinctly.

The first thing to request from a stream location, once you have a fragment
and that you'll need early is TigerTree data for it: it is another
synchronized stream, except that its bitrate is very low compared to the
actual stream data (so that the smallest fragment from the TigerTree stream
can span many fragments from the actual data stream, over a long time
interval, such as 5 to 15 seconds of data stream).

Ideally, each location found in the overlay network (or from the first
location found in a Gnutella query hit during service discovery) should be
able to deliver its TigerTree data stream. So it will deliver both a data
stream and a TigerTree stream using the same timestamp reference. This means
that you don't need a separate URN for the TigerTree stream: it willthen be
necessarily the same (and it will simplify a lot the protocol as you won't
pollute the overlay network to locate other sources for the TigerTree
streams).

We need a standard way to compress the positions within a precise stream, it
is necessarily a compound object: 
- the timestamp encoded as a difference in seconds since the service's
reference epoch, possibly combined with the frame number (which is also
unique within each actual data stream that participates to the service).
- the byte location, relative to the start of the frame.


Note that the frame rate is fixed, like the time rate in seconds. Frames are
then computable subunits of the second. The smallest frame rate is 1 per
second. However thesmallest fragment you'll get andthat you should make
available to the service from the network will be 1 frame (not 1 second).
Consequently, TigerTree data will be computed so that it won't split a
frameinto two separate frames (but it will be allowed to split the data that
makes each second between two frames).
- A frame does not necessarily has a constant length in bytes: with a VBR
codec, frames will be variable in size. That's why the frame rate should not
be too high (above 100 frames per second, it will be excessive and not
necessary for making the stream usable)
- because each service URN uses its own time reference (epoch) you are not
restricted to transport only live events; you could stream as well any file
that uses a streamable format in which you can define a time position and
frame position.
- when you start downloading from a stream, you don't know from which
timestamp or frame number the first fragment you'll get will start: you
first request a fragment starting from timestamp 0 and frame 0 (the
service's epoch) and the uploader will deliver you the oldest fragment that
is after this timestamp and frame number.
- the downloader should be able to wait for enough locations for some
limited time, in order to find the location that has the oldest fragment,
and then it will be able to play or record the stream.
- for origin coders, it should be simple to determine the epoch: either it
knows that it participates to an already existing service URN, so it will
reuse its epoch, or it should choose some simple to predict but still
arbitrary epoch (such as the POSIX epoch for "absolute" time) if the stream
is "live", or simply timestamp 0 for recorded streams, detached from the
live stream. If you want to keep the recorded stream linked to the live
stream from which is comes, you must keep the associated epoch (otherwise
you'll need a new service URN for the detached stream).

To summarize the structure:
* a service URN is what will be searched and found using Gnutella as the
service discovery protocol (where searches are performed by name/metadata)
* it is composed of one or several streams synchronized to the same epoch
* each data stream in this service has its own frame rate, but not
necessarily the same bitrate
* each stream has its own specific metadata for specifying notably its media
type (including the codec type), and its frame rate. This is similar to a
file.
* it's up to downloaders to determine which stream to use according to the
metadata ofeachstream, and to play them synchronously.
* a single TigerTree stream is associated to each data stream, it uses the
same URN as the the datastream, and is synchronized with it using the same
timestamp AND the same frame rate.
* it's up to downloaders to determine which stream it wants from the service
(it can select this according to its mediatype and/or frame rate). The
downloader does not have to download all the streams that he may find
associated to the service (but it will assume the existence of a parallel
stream for the TigerTree data, which should be downloaded as soon as
possible)

Now the download mesh:
* it contains only the locations for the same stream, but will NOT mix the
locations from different services and locations from the same service but
different streams.

Now the downloaders:
* they can participate to the service, download from it, and share the
stream to the network using the same service URN.
* when they do it, they can replicate exactly the same stream, orapply a
transform on it to create a new stream for the service, such as reencoding
it with a different, slower, frame rate or encoding rate. If they do that,
to save their upload bandwidth, they will create a new stream with a
distinct stream URN, with its own stream metadata. But they will also need
to generate a new TigerTree stream and they use the same epoch for allowing
synchronization by downstream users.
* If they record the stream for playing it later, they should keep the
initial timestamps within the record.