Re: [IPFIX] RFC5101: time first flow dropped and time last flow dropped
Paul Aitken <[email protected]>
| Newsgroups | gmane.ietf.ipfix |
|---|---|
| Message-ID | <[email protected]> |
Andrew, Brian,
How do you know that the received template is a MPRSO?
- must the fields be exactly in the specified order and no other
combination in order to be a valid MPRSO?
- how do you know which of the observationTime* elements (or
missingObservationWindowEdge elements) is the first and which is the last?
- only from their position in the template (so, field order and
position are crucial)
- by comparing them (so field order and position aren't important)
Suppose a mediator adds, removes, or re-orders the fields.
- does this invalidate the MPRSO?
- how does a mediator know not to do this?
- how does the collector cope if a mediator does?
The change to "missingObservationWindowEdge" is simply a rename of
observationTime. I don't see that it adds anything.
A long basicList isn't the right solution because
- various list lengths mean there are multiple possible templates
making the detection of the MPRSO a little more difficult
ie, it's no longer a fixed template, nor a combination of a few
possibilities, but a great many different possible templates.
- there are two different semantics for the start and end of the
window, so it's actually an ordered list of pairs ie a subTemplateList.
Lists can have zero elements. However, if you want to report that
nothing was lost, the ignored*Count = 0 report should indicate the start
and end of the 100% period. A list length of zero doesn't tell when
stuff was or was not lost - so it's of limited usefulness, unless
perhaps as a total figure? Again, see questions above about what is / is
not a valid MPRSO.
Thanks,
P.
On 27/09/11 18:30, Andrew Feren wrote:
> Hi Brian,
>
> Well stated.
>
> Additional comments inline
>
> On 09/27/2011 12:15 PM, Brian Trammell wrote:
>> Greetings, all,
>>
>> This is a little bit of an abuse of the ordering of IEs, in my
>> opinion, and of the general rule that only IEs, and the scope in
>> which they are defined, carry semantics. Even for templates which are
>> defined within Standards Track RFCs, the records defined by the
>> template should be decipherable without reference to the RFC. Put
>> another way, we should not define magical templates, and this
>> template as proposed looks pretty magical.
> Yes, a black magic sort of magical. We have been discussing this
> proposed change in the office and coming to the same conclusion.
>> Without reading the text of this thread, and following basically
>> everything we've ever said about multiple IEs in a record, I would
>> interpret a record matching this template as follows:
>>
>> For the metering process A and observation domain B:
>> N packets were ignored; these contained a total of M octets.
>> The first observation at this metering process was at time T0.
>> The second observation at this metering process was at time T1.
> This was exactly how I interpreted this template until I read the
> updated draft and saw the bit about "the Collecting Process MUST
> determine which is the oldest and the most recent timestamp in order
> the determine the right semantic ...."
>
>> This is quite far from the stated intention. I would strongly suggest
>> that we not so lightly give up on "templates should be unambiguously
>> understandable" requirement, even if the exception itself lives in
>> the core protocol document.
>>
>> I would suggest, if you want to do a multiple IE solution here, you
>> define something like missingObservationWindowEdge, and have two of
>> them:
>>
>> observationDomainId {scope}
>> meteringProcessId {scope}
>> ignoredPacketTotalCount
>> ignoredOctetTotalCount
>> missingObservationWindowEdge
>> missingObservationWindowEdge
>>
>> This has the advantage that you can have a long basicList of window
>> edges in order to define multiple missing observation windows; each
>> edge pair is a loss window.
> I like this suggestion, and I have a few additional thoughts.
>
> Perhaps the basic list of edge times should either be limited to only
> one pair or the ignored*TotalCount IEs needs to be in a basiclist with
> N entries too.
>
> How about observationWindowEdge instead (no "missing") to be more
> generally useful.
>
> I think a list of time edges also works neatly when sending this
> template at regular intervals (which SHOULD be done). When the
> ignored*TotalCount IEs are zero, an empty list makes sense for edge
> times.
>
>
>> Seconds suffice for these, I think. You might want to do one in
>> milliseconds. But if you have the resources to calculate your packet
>> loss window to nanosecond precision without having the resources to
>> not lose packets, I humbly submit you've picked the wrong engineering
>> tradeoffs. :)
> Unless window edge isn't defined specifically for "missing" values.
> Then the higher precision timestamps might have some use.
>
> -Andrew
>>
>> Best regards,
>>
>> Brian
>>
>> On Sep 23, 2011, at 11:07 PM, Andrew Feren wrote:
>>
>>> Yes, this makes sense.
>>>
>>> I gave this some thought and I don't have a better suggestion. I
>>> also agree that even though the flow[Start|End]* collection of IEs
>>> conveniently has one IE for both first and last, observationTime* is
>>> the more correct choice here.
>>>
>>> -Andrew
>>>
>>> On 09/22/2011 03:54 AM, Benoit Claise wrote:
>>>> Hi Andrew,
>>>>> Hi Benoit,
>>>>>
>>>>> I had the same thought about using
>>>>>
>>>>> 322 observationTimeSeconds
>>>>> 323 observationTimeMilliseconds
>>>>> 324 observationTimeMicroseconds
>>>>> 325 observationTimeNanoseconds
>>>>>
>>>>> The problem is that you need two timestamps in the same record
>>>>> (first and last). How does that work with the above?
>>>> Excellent question. We debated this question with Paul Aitken
>>>> recently.
>>>> Let me rephrase the question slightly differently: how does the
>>>> collector determine that this is a "The Metering Process
>>>> Reliability Statistics Options Template", and understands the
>>>> semantic of the two counters?
>>>> Well. Two solutions:
>>>> 1. we duplicate all the IEs to have the exact correct semantic.
>>>>
>>>> ObservationTime[packet|flow][first|last][Seconds|Milliseconds|Microseconds|Nanoseconds]
>>>> ... and potentially some more such as pre|post
>>>> This brings multiple new dimensions to IE and will lead to an
>>>> explosion of IEs if we want to cover all the case
>>>> 2. the collector tests the possible combination of IE in the
>>>> Options Template Record.
>>>> For example, if the collector sees such an Options Template
>>>> Record, it knows that this a "The Metering Process Reliability
>>>> Statistics Options Template"
>>>> observationDomainId (scope)
>>>> meteringProcessId (scope)
>>>>
>>>> ignoredPacketTotalCount
>>>> ignoredOctetTotalCount
>>>> observationTimeSeconds | observationTimeMilliseconds |
>>>> observationTimeMicroseconds | observationTimeNanoseconds
>>>> observationTimeSeconds | observationTimeMilliseconds |
>>>> observationTimeMicroseconds | observationTimeNanoseconds
>>>>
>>>>
>>>> While the solution 2 is not perfect, this was the chosen track.
>>>>
>>>> Does it make sense?
>>>>
>>>> Regards, Benoit
>>>>> -Andrew
>>>>>
>>>>> On 09/21/2011 07:11 PM, Benoit Claise wrote:
>>>>>> Hi Gerhard,
>>>>>>
>>>>>> Thanks for your reply.
>>>>>> See in line.
>>>>>>> Hi Benoit,
>>>>>>>
>>>>>>> See comments inline.
>>>>>>>
>>>>>>> On 20.09.2011 16:19, Benoit Claise wrote:
>>>>>>>> Dear all,
>>>>>>>>
>>>>>>>> [this email addresses
>>>>>>>> draft-claise-ipfix-protocol-rfc5101bis-00.txt, TO DO -> "time
>>>>>>>> first flow dropped" and "time last flow dropped"
>>>>>>>> inconsistency. See the discussion on the list.]
>>>>>>>>
>>>>>>>> Paul Aitken discovered this issue.,
>>>>>>>>
>>>>>>>> From [RFC5101], section 4.3. The Exporting Process Reliability
>>>>>>>> Statistics Option Template
>>>>>>>>
>>>>>>>> time first flow dropped
>>>>>>>> The timestamp of the first Flow was
>>>>>>>> dropped by
>>>>>>>> the Metering Process. For this
>>>>>>>> timestamp, any
>>>>>>>> of the "flowStart" timestamp Information
>>>>>>>> Elements flowStartMilliseconds,
>>>>>>>> flowStartMicroseconds,
>>>>>>>> flowStartNanoseconds, and
>>>>>>>> flowStartDeltaMicroseconds can be used.
>>>>>>>>
>>>>>>>> time last flow dropped
>>>>>>>> The timestamp of the last IP packet
>>>>>>>> that was
>>>>>>>> ignored by the Metering Process. For
>>>>>>>> this
>>>>>>>> timestamp, any of the "flowEnd" timestamp
>>>>>>>> Information Elements flowEndMilliseconds,
>>>>>>>> flowEndMicroseconds,
>>>>>>>> flowEndNanoseconds, and
>>>>>>>> flowEndDeltaMicroseconds can be used.
>>>>>>>>
>>>>>>>>
>>>>>>>> Firstly, these definitions are inconsistent since the names and
>>>>>>>> the first definition say "flow" while the second definition
>>>>>>>> says "IP packet". Obviously "IP packet" != "flow" :-o
>>>>>>>>
>>>>>>>> Secondly, "The timestamp of the first Flow was dropped by the
>>>>>>>> Metering Process." is bad English: at least it's missing "that".
>>>>>>>>
>>>>>>>> Thirdly, the section 4.2 doesn't take into account a metering
>>>>>>>> process id. Indeed, what if we have multiple caches on the
>>>>>>>> exporter?
>>>>>>>>
>>>>>>>> Here is a proposal for 4.2 and 4.3. The underlined parts are new
>>>>>>>>
>>>>>>>> 4.2. The Metering Process Reliability Statistics Option Template
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> The Metering Process Reliability Option Template specifies the
>>>>>>>> structure of a Data Record for reporting lack of
>>>>>>>> reliability in the
>>>>>>>> Metering Process. It SHOULD contain the following Information
>>>>>>>> Elements that are defined in [
>>>>>>>> RFC5102
>>>>>>>> ]:
>>>>>>>>
>>>>>>>> observationDomainId
>>>>>>>> An identifier of an Observation
>>>>>>>> Domain that
>>>>>>>> is locally unique to the Exporting
>>>>>>>> Process.
>>>>>>>> This Information Element MUST be
>>>>>>>> defined as a
>>>>>>>> Scope Field.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> meteringProcessId (*)
>>>>>>>> The identifier of the Metering
>>>>>>>> Process for
>>>>>>>> which lack of reliability is
>>>>>>>> reported. There
>>>>>>>> This Information Element MUST be
>>>>>>>> defined as a
>>>>>>>> Scope Field.
>>>>>>>>
>>>>>>>>
>>>>>>>> ignoredPacketTotalCount
>>>>>>>> The total number of IP packets that
>>>>>>>> the
>>>>>>>> Metering Process did not process.
>>>>>>>>
>>>>>>>> ignoredOctetTotalCount
>>>>>>>> The total number of octets in
>>>>>>>> observed IP
>>>>>>>> packets that the Metering Process
>>>>>>>> did not
>>>>>>>> process.
>>>>>>>>
>>>>>>>> time first
>>>>>>>> packet
>>>>>>>> ignored
>>>>>>>> The timestamp of the first IP
>>>>>>>> packet that was
>>>>>>>> ignored by the Metering Process.
>>>>>>>> For this
>>>>>>>> timestamp, any of the "flowStart"
>>>>>>>> timestamp
>>>>>>>> Information Elements
>>>>>>>> flowStartMilliseconds,
>>>>>>>> flowStartMicroseconds,
>>>>>>>> flowStartNanoseconds,
>>>>>>>> and flowStartDeltaMicroseconds can
>>>>>>>> be used.
>>>>>>>>
>>>>>>>> time last
>>>>>>>> packet
>>>>>>>> ignored
>>>>>>>> The timestamp of the last IP packet
>>>>>>>> that was
>>>>>>>> ignored by the Metering Process.
>>>>>>>> For this
>>>>>>>> timestamp, any of the "flowEnd"
>>>>>>>> timestamp
>>>>>>>> Information Elements
>>>>>>>> flowEndMilliseconds,
>>>>>>>> flowEndMicroseconds,
>>>>>>>> flowEndNanoseconds, and
>>>>>>>> flowEndDeltaMicroseconds can be used.
>>>>>>>>
>>>>>>> I do not find these IEs in RFC5102.
>>>>>> Which ones? flowEndMilliseconds, flowEndMicroseconds,
>>>>>> flowEndNanoseconds, and flowEndDeltaMicroseconds
>>>>>> There are in http://www.iana.org/assignments/ipfix/ipfix.xml
>>>>>> However, re-reading
>>>>>> time first packet
>>>>>> ignored
>>>>>> The timestamp of the first IP packet
>>>>>> that was
>>>>>> ignored by the Metering Process. For
>>>>>> this
>>>>>> timestamp, any of the "flowStart"
>>>>>> timestamp
>>>>>> Information Elements
>>>>>> flowStartMilliseconds,
>>>>>> flowStartMicroseconds,
>>>>>> flowStartNanoseconds,
>>>>>> and flowStartDeltaMicroseconds can be
>>>>>> used.
>>>>>>
>>>>>> time last
>>>>>> packet
>>>>>> ignored
>>>>>> The timestamp of the last IP packet
>>>>>> that was
>>>>>> ignored by the Metering Process. For
>>>>>> this
>>>>>> timestamp, any of the "flowEnd"
>>>>>> timestamp
>>>>>> Information Elements
>>>>>> flowEndMilliseconds,
>>>>>> flowEndMicroseconds,
>>>>>> flowEndNanoseconds, and
>>>>>> flowEndDeltaMicroseconds can be used.
>>>>>>
>>>>>> And http://www.iana.org/assignments/ipfix/ipfix.xml for the
>>>>>> "flowEnd" timestamp IE,
>>>>>> I now believe that we should be using this series of IEs
>>>>>> 322 observationTimeSeconds
>>>>>> 323 observationTimeMilliseconds
>>>>>> 324 observationTimeMicroseconds
>>>>>> 325 observationTimeNanoseconds
>>>>>>
>>>>>>>> 4.3. The Exporting Process Reliability Statistics Option Template
>>>>>>>>
>>>>>>>>
>>>>>>>> The Exporting Process Reliability Option Template specifies
>>>>>>>> the
>>>>>>>> structure of a Data Record for reporting lack of
>>>>>>>> reliability in the
>>>>>>>> Exporting process. It SHOULD contain the following
>>>>>>>> Information
>>>>>>>> Elements that are defined in [
>>>>>>>> RFC5102
>>>>>>>> ]:
>>>>>>>>
>>>>>>>> Exporting Process ID
>>>>>>>> The identifier of the Exporting
>>>>>>>> Process for
>>>>>>>> which lack of reliability is
>>>>>>>> reported. There
>>>>>>>> are three Information Elements
>>>>>>>> specified in
>>>>>>>> [
>>>>>>>> RFC5102
>>>>>>>> ] that can be used for this purpose:
>>>>>>>> exporterIPv4Address,
>>>>>>>> exporterIPv6Address, or
>>>>>>>> exportingProcessId. This Information
>>>>>>>> Element
>>>>>>>> MUST be defined as a Scope Field.
>>>>>>>>
>>>>>>>> notSentFlowTotalCount
>>>>>>>> The total number of Flows that were
>>>>>>>> generated by
>>>>>>>> the Metering Process and dropped by
>>>>>>>> the Metering
>>>>>>>> Process or by the Exporting Process
>>>>>>>> instead of
>>>>>>>> being sent to the Collecting Process.
>>>>>>>>
>>>>>>>> notSentPacketTotalCount
>>>>>>>> The total number of packets in Flow
>>>>>>>> Records that
>>>>>>>> were generated by the Metering Process
>>>>>>>> and
>>>>>>>> dropped by the Metering Process or by the
>>>>>>>> Exporting Process instead of being
>>>>>>>> sent to the
>>>>>>>> Collecting Process.
>>>>>>>>
>>>>>>>> notSentOctetTotalCount
>>>>>>>> The total number of octets in packets
>>>>>>>> in Flow
>>>>>>>> Records that were generated by the
>>>>>>>> Metering
>>>>>>>> Process and dropped by the Metering
>>>>>>>> Process or
>>>>>>>> by the Exporting Process instead of
>>>>>>>> being sent
>>>>>>>> to the Collecting Process.
>>>>>>>>
>>>>>>>> time first flow dropped
>>>>>>>>
>>>>>>>> The time at which the first Flow Record
>>>>>>>> was dropped by
>>>>>>>> the
>>>>>>>> Exporting Process.
>>>>>>>> For this timestamp, any
>>>>>>>> of the "flowStart" timestamp Information
>>>>>>>> Elements flowStartMilliseconds,
>>>>>>>> flowStartMicroseconds,
>>>>>>>> flowStartNanoseconds, and
>>>>>>>> flowStartDeltaMicroseconds can be used.
>>>>>>>>
>>>>>>>> time last flow dropped
>>>>>>>>
>>>>>>>> The time at which the last Flow Record was
>>>>>>>> dropped by the Exporting Process
>>>>>>>> . For this
>>>>>>>> timestamp, any of the "flowEnd" timestamp
>>>>>>>> Information Elements flowEndMilliseconds,
>>>>>>>> flowEndMicroseconds,
>>>>>>>> flowEndNanoseconds, and
>>>>>>>> flowEndDeltaMicroseconds can be used.
>>>>>>>>
>>>>>>> Again, I do not find these IEs in RFC5102.
>>>>>> Same remark, I believe we should be using for the two previous IEs
>>>>>> 322 observationTimeSeconds
>>>>>> 323 observationTimeMilliseconds
>>>>>> 324 observationTimeMicroseconds
>>>>>> 325 observationTimeNanoseconds
>>>>>> Regards, Benoit.
>>>>>>>> The Exporting Process SHOULD export the Data Record
>>>>>>>> specified by the
>>>>>>>> Exporting Process Reliability Statistics Option Template on
>>>>>>>> a regular
>>>>>>>> basis or based on some export policy. This periodicity or
>>>>>>>> export
>>>>>>>> policy SHOULD be configurable.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> (*) regarding the meteringProcessId, we were hesitating between
>>>>>>>> the meteringProcessId and the cache id
>>>>>>>> 1. there is a meteringProcessId in IPFIX IANA while there is no
>>>>>>>> cache id
>>>>>>> meteringProcessId should do the job.
>>>>>>>
>>>>>>>> 2. if the IPFIX exporter is configured from [IPFIX-CONF], what
>>>>>>>> should be the value in meteringProcessId?
>>>>>>>> [IPFIX-CONF] doesn't mention the meteringProcessId, but a
>>>>>>>> cache name.
>>>>>>>> From figure 1 and 2, it seems that there is a one to one
>>>>>>>> matching between the cache and the metering process.
>>>>>>>> If this is the case, a solution could be to add a
>>>>>>>> meteringProcess inside the cache in the figure 12 in
>>>>>>>> [IPFIX-CONF] below
>>>>>>> See reply to Pauls mail.
>>>>>>>
>>>>>>> Regards,
>>>>>>> Gerhard
>>>>>>>
>>>>>>>> 4.3. Cache Class
>>>>>>>>
>>>>>>>>
>>>>>>>> +-----------------------------------+
>>>>>>>> | Cache |
>>>>>>>> +-----------------------------------+ 1
>>>>>>>> +------------------+
>>>>>>>> | name |<>--------|
>>>>>>>> immediateCache/ |
>>>>>>>> | dataRecords {readOnly} | |
>>>>>>>> timeoutCache/ |
>>>>>>>> | cacheDiscontinuityTime {readOnly} | |
>>>>>>>> naturalCache/ |
>>>>>>>> | | |
>>>>>>>> permanentCache |
>>>>>>>> | |
>>>>>>>> +------------------+
>>>>>>>> | |
>>>>>>>> | | 0..*
>>>>>>>> +------------------+
>>>>>>>> | |--------->|
>>>>>>>> ExportingProcess |
>>>>>>>> +-----------------------------------+
>>>>>>>> +------------------+
>>>>>>>>
>>>>>>>> Figure 12: Cache class
>>>>>>>>
>>>>>>>> What do you think? Do you see another way?
>>>>>>>>
>>>>>>>> Regards, Paul& Benoit.
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> IPFIX mailing list
>>>>>>>>
>>>>>>>> [email protected]
>>>>>>>> https://www.ietf.org/mailman/listinfo/ipfix
>>>>>>
>>>>>> _______________________________________________
>>>>>> IPFIX mailing list
>>>>>>
>>>>>> [email protected]
>>>>>> https://www.ietf.org/mailman/listinfo/ipfix
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> IPFIX mailing list
>>>>>
>>>>> [email protected]
>>>>> https://www.ietf.org/mailman/listinfo/ipfix
>>> _______________________________________________
>>> IPFIX mailing list
>>> [email protected]
>>> https://www.ietf.org/mailman/listinfo/ipfix
>
> _______________________________________________
> IPFIX mailing list
> [email protected]
> https://www.ietf.org/mailman/listinfo/ipfix
_______________________________________________
IPFIX mailing list
[email protected]
https://www.ietf.org/mailman/listinfo/ipfix