Re: [IPFIX] FW: UDP Sequence Numbers rfc 5101

Brian,

I agree, except for "Template message loss can be deduced by observing 
Data Sets covered by unreceived Templates in any case."

This isn't true in the case of template redefinition, specifically 
mentioned by Steve.

ie, a collector will be unaware that a redefined template has been lost, 
especially if the new and old data records are the same size - although 
records might not decode sensibly. However I'm not sure that many 
collectors come with AI to determine whether records are sensible.

In the worst case, the new template will specify records with a length 
that's a multiple of the old record length, so messages will 
intermittently appear to be too short.

As ever, template reuse over unreliable transport is best avoided.

P.



On 03/04/13 20:53, Brian Trammell wrote:
> hi Steve, all,
>
> On 4 Apr 2013, at 2:49, "Steve Nash" <[email protected]> wrote:
>
>> Paul
>> Two things:
>> 1/ Thanks for the explanation.  The assessment of relative importance of
>> template loss v count of data records is clearly case depended, but
>> reasonable.
>> 2/ Sentence three remains incorrect  to my mind.  It is a Recommendation
>> that is impossible to fulfil .
> While it is technically correct that it is Data Records that are counted by the Sequence Number and not Messages, it is Messages that are the unit of loss over UDP or partially-reliable SCTP, therefore the signal available from sequence number analysis is a loss-of-message signal.
>
> It is true that you can't deduce the loss of UDP template-only messages by looking at IPFIX Message Header Sequence Numbers. But you can't do that in V9 either. There's no information about the contents of lost Export Packets in V9. Template message loss can be deduced by observing Data Sets covered by unreceived Templates in any case.
>
>> As it stands the sentence leads the reader to believe that IPFIX over UDP
>> provides greater reliability than it actually does.
> In the context of the document and the documents around it (see e.g. 5153 section 6.2, 5101bis section 10.1 para 4 and section 10.3) I can't see how this is the case.
>
> Anything-over-UDP is unreliable, full stop, and we should not put a lot of effort in to pretending it could be.
>
> Regards,
>
> Brian
>
>> -----Original Message-----
>> From: Paul Aitken [mailto:[email protected]]
>> Sent: 03 April 2013 11:42
>> To: [email protected]
>> Cc: [email protected]
>> Subject: Re: [IPFIX] FW: UDP Sequence Numbers rfc 5101
>>
>> Steve,
>>
>> The sequence number change was a deliberate decision in RFC 5101, so that
>> collectors can use the sequence numbers to determine (and thus
>> report) the number of lost data records.
>>
>> This is an improvement of RFC 3954, where the sequence number only counted
>> export packets. In this case it's impossible to determine how many data
>> records were lost, ie the amount of loss is unquantifiable.
>> The packets could have contained templates and/or data records; we simply
>> don't know.
>>
>> Neither of these sequence numbers helps to specifically determine whether a
>> template has been lost. 3954 sequence numbers identify packet loss without
>> indicating whether it's template or data record. 5101 sequence numbers
>> identify data record loss only.
>>
>> Identifying template loss would require an additional template sequence
>> number.
>>
>> If loss is a big issue in your network, don't export over UDP. If you must
>> export over UDP, and loss is an issue, then don't redefine templates on the
>> fly.
>>
>> In my experience templates are generally static (defined once and never
>> withdrawn or redefined), and limited to a few 10's in number. However, there
>> are 65280 templates available. Defining these incrementally at 1000/sec with
>> a 1-minute lifetime, or at 100/sec with a 10-minute lifetime, ensures that
>> no template will ever be redefined within its lifetime - although I can't
>> think of any use case that requires (re)defining templates that quickly.
>>
>> P.
>>
>>
>> On 03/04/13 10:52, Steve Nash wrote:
>>> The third sentence remains incorrect.  Only IPFIX Data Records are
>>> protected by sequence number .  "IPFIX messages" in general are not
>>> protected (even templates).
>>> This is a retrograde step from rfc3954.
>>> It makes use of IPFIX for flow reporting over UDP less tolerant than
>>> rfc 3954.  The unrecognised loss of a template change message could
>>> result in significant misinterpretation of data, which is worse than
>>> just recognised data loss.
>>>
>>> The second sentence seems redundant, as UDP is no longer different to
>>> TCP and SCTP.  You can avoid any mention of transport protocol here.
>>>
>>> Regards
>>> Steve
>>>
>>>
>>> -----Original Message-----
>>> From: Brian Trammell [mailto:[email protected]]
>>> Sent: 25 March 2013 00:32
>>> To: [email protected]
>>> Cc: [email protected]
>>> Subject: Re: [IPFIX] UDP Sequence Numbers rfc 5101
>>>
>>> Hi, Steve,
>>>
>>> This is an error in RFC 5101; the intent of the Sequence Number is
>>> that it be Observation Domain (and in the case of SCTP, Stream) scoped
>>> (i.e., you are correct that transport independence, to the extent
>>> possible, is the intent). The language in the present revision of
>>> 5101bis still contains the same error. Thanks for the catch!
>>>
>>> I would suggest we fix this by replacing the paragraph in question as
>>> follows:
>>>
>>> OLD:
>>>
>>>     The Collecting Process SHOULD deduce the loss and reordering of IPFIX
>>>     Data Records by looking at the discontinuities in the IPFIX Sequence
>>>     Number.  In the case of UDP, the IPFIX Sequence Number contains the
>>>     total number of IPFIX Data Records sent for the UDP Transport Session
>>>     prior to the receipt of this IPFIX Message, modulo 2^32.  A Collector
>>>     SHOULD detect out-of-sequence, dropped, or duplicate IPFIX Messages
>>>     by tracking the Sequence Number.
>>>
>>> NEW:
>>>
>>>     The Collecting Process SHOULD deduce the loss and reordering of IPFIX
>>>     Data Records by looking at the discontinuities in the IPFIX Sequence
>>>     Number.  In the case of UDP, the IPFIX Sequence Number contains the
>>>     total number of IPFIX Data Records sent for the Observation Domain in
>> the
>>>     current UDP Transport Session prior to the receipt of this IPFIX
>>> Message,
>>>
>>>     modulo 2^32.  A Collector SHOULD detect out-of-sequence, dropped, or
>>>     duplicate IPFIX Messages by tracking the Sequence Number.
>>>
>>> The document is presently in Publication Requested state, so I think
>>> this should be placed as an RFC editor note on the document. WG: any
>>> comments thereon?
>>>
>>> Best regards,
>>>
>>> Brian
>>>
>>> On 22 Mar 2013, at 0:45, Steve Nash <[email protected]> wrote:
>>>
>>>> Please consider the wording of the paragraph on Sequence Numbers in
>>> Section 3.1.
>>>>        Incremental sequence counter modulo 2^32 of all IPFIX Data Records
>>>>        sent in the current stream from the current Observation Domain by
>>>>        the Exporting Process. Each SCTP Stream counts sequence numbers
>>>>        separately, while all messages in a TCP connection or UDP
>>>>        transport session are considered to be part of the same stream.
>>>> The phrase I have put in italics is depended on Transport protocol.
>>>> Section 10.3.2 Reliability says:
>>>>
>>>> In the case of UDP, the IPFIX Sequence Number contains the total
>>>> number of IPFIX Data Records sent for the UDP Transport Session prior
>>>> to the receipt of this IPFIX Message
>>>>
>>>> This is NOT dependent on the ObservationDomain.  Nothing prohibits
>>>> the
>>> Exporting Process from using the same Transport Session for multiple
>>> Observation Domains.  But in UDP the Sequence number is for the
>>> Transport session, whereas in TCP and SCTP each Observation Domain has
>>> an independent sequence on the same  (and on each) Transport Session.
>>>> The term 'Stream' seems to take different meanings in IPFIX,
>>>> depending on
>>> the transport protocol.
>>>> This is not consistent with IPFIX being  "Transport protocol independent
>> "
>>> (Section 10 first para).
>>>> Regards
>>>> Steve Nash
>>>> [email protected]
>>>> UK
>>>> _______________________________________________
>>>> IPFIX mailing list
>>>> [email protected]
>>>> https://www.ietf.org/mailman/listinfo/ipfix
>>> _______________________________________________
>>> IPFIX mailing list
>>> [email protected]
>>> https://www.ietf.org/mailman/listinfo/ipfix
>> _______________________________________________
>> IPFIX mailing list
>> [email protected]
>> https://www.ietf.org/mailman/listinfo/ipfix

_______________________________________________
IPFIX mailing list
[email protected]
https://www.ietf.org/mailman/listinfo/ipfix