Re: Should FLUTE File Aggregation I-D be a working group item?

Hi Rod,

I think fioe aggregation is basically a useful thing to consider as
the motivation clearly points out.  In Vancouver, I raised the point
about re-inventing gzip.  The following statement in the motivation
section makes me a bit nervous:

    Another motivation for having an object aggregation scheme compared
    to a basic archive based solution (e.g. tarball), is that no extra
    transformation (i.e. archive creation or extraction) is required at
    either the FLUTE sender or receiver.  Everything is managed
    automatically by the transport mechanism according to transport-
    specific optimizations and can be transparent to upper applications
    (i.e. built on top of FLUTE), or enhanced by application hints on
    file relationships, without breaking the basic semantics of FLUTE
    sessions.

Yes, you can always push functionality down a layer (which still means
that you have to perform the function but you just abstract from its
details for the application and limit its amount of work and the
flexibility in applying this function).  But you should carefully
ask yourself how far you want to take this.  Will we see
semantically more complex relationships tomorrow?  Compression
of data?  ...?  How do you deal with partially received objects?
How complex does the API to your transport get?

There is a certain number of things that only the application can do
right by itself.  I would like to see a clear borderline drawn
of what shall go into the FLUTE transport and what should stay up
at the application layer and then we should make sure that the
extensions we define now are going to be orthogonal to other
features we may want to add to the transport later and that they
are easily usable by the application for what it may want to do.

(Remember that we are not building "systems" in the Transport Area
and that we do not necessarily have to build application libraries
either.  So, how much is needed for interoperability in FLUTE?)

Cheers,
Joerg

> Hi Mark et al.
> 
> Thanks for the support
> 
> <I'll take time to consider the AO semantic suggestions - an email for
> another day> 
> 
> 
>>Some other minor comments:
> 
> 
>>- I don't think we need to worry about backwards compatibility 
>>with FLUTE receivers compliant to the experimental version 
>>which did not have element extensibility in the FDT XML - this 
>>was a mistake and the 'experimental' phase is exactly supposed 
>>to allow us to find and correct such mistakes.
> 
> 
> Now the update FLUTE is practically ready (anticipate one last I-D
> update), I agree.
> 
> The AO I-D previously the use of non-TOI=0 FDTs to help  with Exp
> backwards compatibility. Non-TOI=0 FDTs are allowed anyway, but we need
> not explain that this is useful in handling the experimental FDT
> extensibility issue.
> 
> 
>>- the draft includes some text attaching semantics to TOI 
>>ordering - something we removed from FLUTE in favour of FDT 
>>Instance IDs. We should stick with the FLUTE principle that 
>>later version of a file is signalled by using a different TOI 
>>in an FDT with a larger FDT Instance ID - it doesn't matter 
>>whether the TOI value is greater or less than the previous one.
> 
> 
> Note, for FLUTE there is no such semantic. The client has no gauntees
> about FDT Instance numbers, no garuntees about TOI numbers.
> 
> Generally, using TOI and 'FDT Instance number' values for this has
> similar issues (assuming wrap-around is handled smoothly, higher/later
> values represent later versions, gaps in the range of number are
> feasible, FDT Instances may never be received). TOI keeps semantics with
> the file declaration; FDT requires inheritence from FDT Instance to
> files described for that Instance. Implemantation-wise there's little
> difference between the two: probably the server will be incrementing FDT
> Instance numbers and not describing out-of-date TOIs on new Instances;
> probably the server will use higher TOI values for later versions; the
> client must store TOI with fileURI. There is no requierment that the
> client stores FDT Instance numbers.
> 
> Quick FLUTE quote:
> 
>    A receiver of the file delivery session keeps an FDT database for
>    received file description entries.  The receiver maintains the
>    database, for example, upon reception of FDT Instances.  Thus, at any
>    given time the contents of the FDT database represent the receiver's
>    current view of the FDT of the file delivery session.  Since each
>    receiver behaves independently of other receivers, it SHOULD NOT be
>    assumed that the contents of the FDT database are the same for all
>    the receivers of a given file delivery session.
> 
>    Since FDT database is an abstract concept, the structure and the
>    maintaining of the FDT database are left to individual
>    implementations and are thus out of scope of this specification.
> ...
>    *  The receiver SHOULD NOT use a received FDT Instance to interpret
>       packets received beyond the expiration time of the FDT Instance.
> 
> i.e. the expiry time of file description should be kept with other file
> data, but FDT Instance ID is up to the implementation.
> 
> Forcing clients to additional store this value is not much trouble - but
> it is _not_ "the FLUTE principle that later version of a file is
> signaled by using a different TOI in an FDT with a larger FDT Instance
> ID". Which is exactly why the AO draft offers the simplest solution to
> this issue. If we make the assumptions of "well-behaved simpple FLUTE
> servers" above, there really is nothing between FDT Instance ID and TOI
> to be used for this purpose. The reason I give all this background is
> that this doesn't hold for some obvious AO usage...
> 
> For a physically aggregated object (e.g. very large) with some small
> updates (e.g. 2 file versions are new from 10,000 files). The server
> will use TOI (i) for the initial full object files descriptions. It is
> highly likely that the big aggregated object will continue to be
> transmitted for some time and with only minimal updates it is probably
> more bandwidth efficient to send the symbols for the original object and
> the few (e.g. 2) new file versions seperates (more efficient due to,
> e.g. fountain, symbols of earlier reception still being useful to all
> receivers regarless of complete, partial or not-started receiption
> status). So the server sends FDT (ii) with the new TOIs for the files
> which have been updated. <At this point using either TOI or FDT Instance
> ID would work>. However, some objects of FDT Insatnce (i) are now
> changed/removed so that the server need to describe the slighlty old
> large aggregated object on TOI (iii). All recievers throw away the new
> files (even though the TOI is higher) as the old files are described in
> a later FDT Instance.
> 
> FLUTE is specifically designed to allow this ebhaviour and make no
> assumption the receivers FDT/file database state and no assumption on
> reception of individual FDT Instances.
> 
> Using TOI for this is simple and works. FDT Instance ID fails. Hence,
> the mention of TOI for this purpose in the I-D. (Originally it was start
> with TOI=1 and increment by 1, but that's overkill as just 'higher
> value' is needed for subseqent versions of a file (fileURI) instance)
> 
> (I've been trying to explain this for a while - but it's not important
> in 3GPP rel6, and I'm not up-to-date with DVB or OMA to see if
> something's fractured there - so maybe this is the first time I've
> explained by mail.)
> 
> 
>>- the draft mentions that one might only consider file 
>>aggregation when the aggregate is less than a single source 
>>block - I think it is simpler and has no disadvantages if you 
>>decouple the source blocking (which is a feature of the FEC 
>>scheme chosen) from the file aggregation. If files belong 
>>together in an aggregate then they should be put together and 
>>then if the FEC code needs to split that data into blocks this 
>>is ok and does not result in any worse performance so long as 
>>it correctly interleaves the packets from the different blocks.
> 
> 
> Exactly.
> 
> The I-D does not mean to bind AO and FEC blocking. The discussion is for
> implementation guidence: for files much greater than the block size
> there's no FEC benefit in physical aggregation; for the composite files
> being less than the block size it always makes sense to aggregate for
> robustness; inbetween it's a compromise; the I-D absolutely does not
> impose any FEC instance retraints and so it's implementation specific to
> optomise complexity and performance in this aspect. The guidence is
> valuable. If we inadvertantly made normative specification on this, it
> was an error to be corrected.
> 
> Also, the logical grouping between objects can be used to get the
> "belongs together" semantics for multiple AOs - e.g. where partial
> recovery is undesirable, or different levels of robustness are
> desirable).
> 
> Cheers, Rod.
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> _______________________________________________
> Rmt mailing list
> [email protected]
> https://www1.ietf.org/mailman/listinfo/rmt
>