Re: Should FLUTE File Aggregation I-D be a working group item?

Hi all,

sorry for my late reply to all your comments. I think quite interesting
issues have been raised.
I will only talk about physical aggregation, since the logical aggregation
scheme seems quite obvious to everyone.

First, what is clear to all of us, is that the object aggregation scheme
should be more than just a gzip or tar of several files.

Therefore, we need to clearly identify the advantages we want and can
provide. From the previous mails I identfied the following features:
I. Partial reception: Extraction of files from partially recovered AO

II. Improved transmission performances (eliminating coupon collector +
better FEC performances)

III. Provide FDT information (MIME-type, URI etc.) for each of the files
that have been aggregated

IV. Backward compatibility:
    This feature can be interpreted in two ways:
      1) Receivers which do not support the aggregation scheme, should be
able to interpret at least the source symbols sent for the files being
aggregated.
      2) Receivers which do not support the aggregation scheme, should be
able to interpret all source and parity symbols of the aggregated object.
I.e the receivers can reconstruct the entire aggregated object, and then
decompose it into the files being aggregated.

V. Incremental update: It should be possible to reuse previously received
symbols of an old aggregated object or of old files, when these files are
reused in a new aggregated object.

VI. Single file transmission: It should be possible to send the files being
aggregated individually.


 The "gzip approach", only supports features II. and IV.2).

The draft presents a solution that supports feature I. II. III. IV.2) and
nothing more.
The MIME format was chosen primarily with issue IV.2) in mind. As mentioned
in your comments it is maybe not the best solution and we may have a look at
some alternatives.

Another proposal that came out of the discussions looks interesting to me:
sending each individual file on a separate TOI and having an aggregated
object that uses another TOI.
The parity symbols produced for the aggregated object are generated based on
all source symbols of the files being aggregated. This means that only
parity symbols are send for the AO.
With that approach the FDT must list the TOIs of the files being aggregated
and their offsets within the aggregated object (maybe some additional
attributes will also be needed).
As mentioned by Mark, the FDT would be much simpler compared to the current
approach proposed in the draft. However, we must clearly think about the
mechanisms used by such an approach.

This new approach should be able to cover the features I. II. III. IV.1) V.
and VI.
The only concern is that receivers which do not support the aggregation
scheme cannot use the parity symbols
of the aggregated object. However I think this is acceptable, since the
sender may produce and send parity symbols for each individual file, if he
suspects having receivers that do not support the aggregation scheme.

I think it is worth investigating this proposal, and I am a volunteer for
doing that. As pointed out by Mark, the resulting draft will be rather
different from the one presently proposed. With regards to the comments in
this thread, I think that the new document should then become a working
group item.

cheers,

Christoph


On 12/2/05, [email protected] <[email protected]> wrote:
>
> This is an excellent point. Basically we do not want to specify
> applications in RMT and vigilance is essential.
>
> (I'll skip over the logical aggregation chatter as the thread is focused
> on physical aggregation).
>
> FLUTE already allows content encoding for transport - great idea taken
> from HTTP. FDTs provide the necessary identification for files and the
> CENC header extension for FDT Instances. As such it is possible to
> "code" (e.g. zlib/gzip) a single file. For doing this to a group of
> files you have but 2 options:
> - gzip each file and transport separately
> - gzip together in an application and reveal only the .zip file & MIME
> type to FLUTE (presumably with no content encoding in this case)
>
> These 2 options are fine in their respective use cases; and note I'm
> using zlib and gzip only to make the discussion clearer - there are
> several other examples which may be relevant (tarball, ...)
>
> The problem is the 2 options force you to lose out on transport
> functionality. You either forget FEC performance (many small files) or
> you forget file delivery (i.e. FLUTE becomes a container shipment
> activity with no knowledge of files).
>
> If you need the atomic file deliver primary function of FLUTE, as well
> as the FEC performance of larger objects then you'll need a 3rd option
> at the transport layer = the physical aggregation technique.
>
> A really exciting part is that although transport grouping binds files
> together (in an all-or-nothing sense); with partial recovery you would
> not only be able to serve all-or-nothing applications. i.e. handling
> this as transport really can improve transport performance in ways that
> "above the FEC" application functions would not.
>
> Cheers, Rod.
>
> PS if gzip provided the right data about files in an external/exportable
> MIME type and allowed partial recovery, then I'd go for simple adoption
> of gzip, but it doesn't - so I think we can either specify this simple
> object aggregation, or we create gzip2 which does have the features we
> need at FLUTE transport. I don't like the idea of inventing gzip either,
> and I'm not too keen on evangelising the adoption of re-invented gzip2
> :)
>
>
>
> >-----Original Message-----
> >From: ext Joerg Ott [mailto: [email protected]]
> >Sent: 28 November, 2005 10:09
> >To: Walsh Rod (Nokia-NRC/Tampere)
> >Cc: [email protected]; [email protected];
> >[email protected]; [email protected]
> >Subject: Re: [Rmt] Should FLUTE File Aggregation I-D be a
> >working group item?
> >
> >Hi Rod,
> >
> >I think fioe aggregation is basically a useful thing to
> >consider as the motivation clearly points out.  In Vancouver,
> >I raised the point about re-inventing gzip.  The following
> >statement in the motivation section makes me a bit nervous:
> >
> >    Another motivation for having an object aggregation scheme compared
> >    to a basic archive based solution (e.g. tarball), is that no extra
> >    transformation (i.e. archive creation or extraction) is required at
> >    either the FLUTE sender or receiver.  Everything is managed
> >    automatically by the transport mechanism according to transport-
> >    specific optimizations and can be transparent to upper applications
> >    (i.e. built on top of FLUTE), or enhanced by application hints on
> >    file relationships, without breaking the basic semantics of FLUTE
> >    sessions.
> >
> >Yes, you can always push functionality down a layer (which
> >still means that you have to perform the function but you just
> >abstract from its details for the application and limit its
> >amount of work and the flexibility in applying this function).
> > But you should carefully ask yourself how far you want to
> >take this.  Will we see semantically more complex
> >relationships tomorrow?  Compression of data?  ...?  How do
> >you deal with partially received objects?
> >How complex does the API to your transport get?
> >
> >There is a certain number of things that only the application
> >can do right by itself.  I would like to see a clear
> >borderline drawn of what shall go into the FLUTE transport and
> >what should stay up at the application layer and then we
> >should make sure that the extensions we define now are going
> >to be orthogonal to other features we may want to add to the
> >transport later and that they are easily usable by the
> >application for what it may want to do.
> >
> >(Remember that we are not building "systems" in the Transport
> >Area and that we do not necessarily have to build application
> >libraries either.  So, how much is needed for interoperability
> >in FLUTE?)
> >
> >Cheers,
> >Joerg
> >
> >> Hi Mark et al.
> >>
> >> Thanks for the support
> >>
> >> <I'll take time to consider the AO semantic suggestions - an
> >email for
> >> another day>
> >>
> >>
> >>>Some other minor comments:
> >>
> >>
> >>>- I don't think we need to worry about backwards compatibility with
> >>>FLUTE receivers compliant to the experimental version which did not
> >>>have element extensibility in the FDT XML - this was a
> >mistake and the
> >>>'experimental' phase is exactly supposed to allow us to find and
> >>>correct such mistakes.
> >>
> >>
> >> Now the update FLUTE is practically ready (anticipate one last I-D
> >> update), I agree.
> >>
> >> The AO I-D previously the use of non-TOI=0 FDTs to help  with Exp
> >> backwards compatibility. Non-TOI=0 FDTs are allowed anyway, but we
> >> need not explain that this is useful in handling the
> >experimental FDT
> >> extensibility issue.
> >>
> >>
> >>>- the draft includes some text attaching semantics to TOI ordering -
> >>>something we removed from FLUTE in favour of FDT Instance IDs. We
> >>>should stick with the FLUTE principle that later version of
> >a file is
> >>>signalled by using a different TOI in an FDT with a larger FDT
> >>>Instance ID - it doesn't matter whether the TOI value is greater or
> >>>less than the previous one.
> >>
> >>
> >> Note, for FLUTE there is no such semantic. The client has no
> >gauntees
> >> about FDT Instance numbers, no garuntees about TOI numbers.
> >>
> >> Generally, using TOI and 'FDT Instance number' values for this has
> >> similar issues (assuming wrap-around is handled smoothly,
> >higher/later
> >> values represent later versions, gaps in the range of number are
> >> feasible, FDT Instances may never be received). TOI keeps semantics
> >> with the file declaration; FDT requires inheritence from FDT
> >Instance
> >> to files described for that Instance. Implemantation-wise there's
> >> little difference between the two: probably the server will be
> >> incrementing FDT Instance numbers and not describing
> >out-of-date TOIs
> >> on new Instances; probably the server will use higher TOI values for
> >> later versions; the client must store TOI with fileURI. There is no
> >> requierment that the client stores FDT Instance numbers.
> >>
> >> Quick FLUTE quote:
> >>
> >>    A receiver of the file delivery session keeps an FDT database for
> >>    received file description entries.  The receiver maintains the
> >>    database, for example, upon reception of FDT Instances.
> >Thus, at any
> >>    given time the contents of the FDT database represent the
> >receiver's
> >>    current view of the FDT of the file delivery session.  Since each
> >>    receiver behaves independently of other receivers, it
> >SHOULD NOT be
> >>    assumed that the contents of the FDT database are the same for all
> >>    the receivers of a given file delivery session.
> >>
> >>    Since FDT database is an abstract concept, the structure and the
> >>    maintaining of the FDT database are left to individual
> >>    implementations and are thus out of scope of this specification.
> >> ...
> >>    *  The receiver SHOULD NOT use a received FDT Instance to
> >interpret
> >>       packets received beyond the expiration time of the FDT
> >Instance.
> >>
> >> i.e. the expiry time of file description should be kept with other
> >> file data, but FDT Instance ID is up to the implementation.
> >>
> >> Forcing clients to additional store this value is not much trouble -
> >> but it is _not_ "the FLUTE principle that later version of a file is
> >> signaled by using a different TOI in an FDT with a larger
> >FDT Instance
> >> ID". Which is exactly why the AO draft offers the simplest
> >solution to
> >> this issue. If we make the assumptions of "well-behaved
> >simpple FLUTE
> >> servers" above, there really is nothing between FDT Instance ID and
> >> TOI to be used for this purpose. The reason I give all this
> >background
> >> is that this doesn't hold for some obvious AO usage...
> >>
> >> For a physically aggregated object (e.g. very large) with some small
> >> updates (e.g. 2 file versions are new from 10,000 files). The server
> >> will use TOI (i) for the initial full object files
> >descriptions. It is
> >> highly likely that the big aggregated object will continue to be
> >> transmitted for some time and with only minimal updates it
> >is probably
> >> more bandwidth efficient to send the symbols for the original object
> >> and the few (e.g. 2) new file versions seperates (more efficient due
> >> to, e.g. fountain, symbols of earlier reception still being
> >useful to
> >> all receivers regarless of complete, partial or not-started
> >receiption
> >> status). So the server sends FDT (ii) with the new TOIs for
> >the files
> >> which have been updated. <At this point using either TOI or FDT
> >> Instance ID would work>. However, some objects of FDT
> >Insatnce (i) are
> >> now changed/removed so that the server need to describe the slighlty
> >> old large aggregated object on TOI (iii). All recievers
> >throw away the
> >> new files (even though the TOI is higher) as the old files are
> >> described in a later FDT Instance.
> >>
> >> FLUTE is specifically designed to allow this ebhaviour and make no
> >> assumption the receivers FDT/file database state and no
> >assumption on
> >> reception of individual FDT Instances.
> >>
> >> Using TOI for this is simple and works. FDT Instance ID
> >fails. Hence,
> >> the mention of TOI for this purpose in the I-D. (Originally it was
> >> start with TOI=1 and increment by 1, but that's overkill as just
> >> 'higher value' is needed for subseqent versions of a file (fileURI)
> >> instance)
> >>
> >> (I've been trying to explain this for a while - but it's not
> >important
> >> in 3GPP rel6, and I'm not up-to-date with DVB or OMA to see if
> >> something's fractured there - so maybe this is the first time I've
> >> explained by mail.)
> >>
> >>
> >>>- the draft mentions that one might only consider file aggregation
> >>>when the aggregate is less than a single source block - I
> >think it is
> >>>simpler and has no disadvantages if you decouple the source blocking
> >>>(which is a feature of the FEC scheme chosen) from the file
> >>>aggregation. If files belong together in an aggregate then
> >they should
> >>>be put together and then if the FEC code needs to split that
> >data into
> >>>blocks this is ok and does not result in any worse
> >performance so long
> >>>as it correctly interleaves the packets from the different blocks.
> >>
> >>
> >> Exactly.
> >>
> >> The I-D does not mean to bind AO and FEC blocking. The
> >discussion is for
> >> implementation guidence: for files much greater than the block size
> >> there's no FEC benefit in physical aggregation; for the
> >composite files
> >> being less than the block size it always makes sense to aggregate for
> >> robustness; inbetween it's a compromise; the I-D absolutely does not
> >> impose any FEC instance retraints and so it's implementation
> >specific to
> >> optomise complexity and performance in this aspect. The guidence is
> >> valuable. If we inadvertantly made normative specification
> >on this, it
> >> was an error to be corrected.
> >>
> >> Also, the logical grouping between objects can be used to get the
> >> "belongs together" semantics for multiple AOs - e.g. where partial
> >> recovery is undesirable, or different levels of robustness are
> >> desirable).
> >>
> >> Cheers, Rod.
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >> _______________________________________________
> >> Rmt mailing list
> >> [email protected]
> >> https://www1.ietf.org/mailman/listinfo/rmt
> >>
> >
> >
>
> _______________________________________________
> Rmt mailing list
> [email protected]
> https://www1.ietf.org/mailman/listinfo/rmt
>
>
>

_______________________________________________
Rmt mailing list
[email protected]
https://www1.ietf.org/mailman/listinfo/rmt