Re: [IPFIX] WG Last Call for draft-ietf-ipfix-information-model-rfc5101bis-03.txt

Hi, Andrew, all,

(copying the list; as this conversation probably belongs there...)

replies inline...

On Dec 8, 2012, at 12:39 AM, Andrew Johnson wrote:
> On 6 Dec 2012, at 21:03, Brian Trammell <[email protected]> wrote:
>> On Dec 6, 2012, at 6:02 PM, Andrew Johnson wrote:
>> 
>>> Hi Brian
>>> 
>>> I'm working my way through the changes to RFC5101.  For the most part I like the clarifications and tweaks that have been made, but I have a few niggles that hopefully you can straighten out for me.  Sorry for suddenly get involved at WGLC, but I'm pretty sure my comments are minor, I'm just missing some of the background to why these things changed.
>> 
>> The more review, and the more comments, the better, and "during WGLC" is still on time, so thanks!
>> 
>>> 1. Will new collectors fail to recognise RFC5101 compliant Metering Process
>>> reports because of the new scope field?  Should the new field be a SHOULD
>>> so that collecting processes have to treat it as optional?  (Alternatively,
>>> "For exporting processes that are only dealing with a single Metering
>>>  Process, the meteringProcessId scope fields MAY be omitted"?)
>>> 
>>> If this sort of thing is allowed because it's clear that the old style
>>> is RFC5101 and the new style is RFCXXXX, then I'm fine with the change.
>> 
>> Hm. I'd submit that any collector that didn't allow for additional fields in a "recognized template" was probably not very well designed, but we don't say that anywhere in the documents. It's clear to me because the specific reporting requirements are exporter-only, and the collectors should be designed in such a way to handle anything an exporter might reasonably throw at them. But maybe we could add some text in these sections to say that these are the minimal IEs; exporters may add other appropriate IEs to these templates as desired. Pre 5101-bis collectors that required exactly those fields (in exactly that order), though, would be out of luck. Personally, I don't find this problematic, but that's just a matter of opinion on my part.
> 
> I don't have a problem with it either, as long is acceptable when updating 5101.  I also agree that collectors should be prepared to accept additional elements in the reports, although I'd always thought it would be additional non-scope extensions providing different stats, rather than additional scope elements changing the specificity of where the stats apply to.
> 

I'll suggest (this is a note to self, more than anything) adding a sentence to each of these templates on this point, and default assumptions to be made at collectors when new scope elements are added.

>>> 2. The new text for the timestamps seems to prohibit interpreting a rolled
>>> over timestamp accurately.  i.e. in 2036 dateTimeMicroseconds and
>>> dateTimeNanoseconds become useless.  NTP seems to be able to cope just
>>> fine with the rollover, and it seems pretty short sighted of us to
>>> design a protocol that won't last another 25 years.
>> 
>> Hm. This depends on the assumptions you make.
> 
> The definition says "It can represent dates beginning between 1 January 1900 and 8 February 2036".  It doesn't say anything about worrying about the era.

True; it's precently implicitly an Era 0 date.

> 
>> In NTPv4, there's an "era" concept, which explicitly handles 64-bit NTP wraparound. We should probably change these definitions to incorporate this concept, stating that the era is implicit, and that near era boundaries, the collector should make a presumption that eras tend to be the current era, where current is defined by the current time at the collector.
> 
> Agreed, the era idea needs to be included in the definition.  In my code I make the assumption that the times are in the past (or very near future in case of clock skews).  Usually it'll be pretty obvious but we need to point out that it's possible.

Adding implicit eras to the NTP types makes them work, and indeed, we could add implicit eras to the other timestamp types, as well, in order to allow more flexible interpretation thereof (maybe a specific subsection on wraparound?). 

>> This breaks horribly for files; if you want data archiving, you need a way to store era number for these timestamps in the file.
> 
> Don't you have any timestamps in message parts that can guide the interpretations.  Maybe you can recommend opening a new file every hundred years or so.

Ah, wasn't concerned with lifetimes (and indeed a clever collector should be able to handle a century-long file by keeping wrap counters on a per-record basis); was more concerned with unambiguous interpretation of files that have been stored for years, decades, or centuries. 

> 
>> This only fixes the issues with NTP timestamps. dateTimeSeconds and Export Time in the header are still only good for another 95 years or so (which is why I would _not_ base the era determination on the Export Time). Sometime between now and February 8, 2106, we should probably define new timestamp types which have representation independent from epoch, with a 64-bit value somewhere for defining the epoch.
> 
> I've pencilled it into my diary for 1st April, 2099.  Are you free?

We've kicked the can down the road for another century, which I think is good enough for me. Can I get back to you in late '98?

>>> 3. In section 8, when sending the withdraw all templates message, should
>>> flowset ID 2 be used to contain the withdrawal of template ID 2?
>>> Similarly, when withdrawing all options templates, does the message
>>> have to be sent in flowset 3?
>> 
>> That's my interpretation of 5101.
> 
> It didn't seem to be explicitly stated, although it's how I'd have chosen to do it.  If everyone would just do it they way I would, we wouldn't even need a standards body :).

So we'll make this explicit in 5101bis as well.

>>> 4. In section 8.2: "Since there is no guarantee of the ordering of
>>> exported IPFIX, Messages across SCTP Streams".  I thought PR-SCTP
>>> allowed explicit ordering using an optional Stream-Sequence field.
>>> Doesn't that mean the SCTP order can be assured for templates?
>>> 
>>> What happened to the requirement to use reliable, in-order delivery?
>> 
>> There's ordering within the stream, and ordering of sending across streams, but you can't guarantee the ordering of processing, especially if the collector is designed to handle streams independently.
> 
> Ah!  I'd missed the across streams bit.  There's an RFC somewhere that recommends against that :)
> 
> 
>>> 5. "Collecting Processes MAY implement a buffer to handle out-of-order
>>> Template management events"
>>> 
>>> It's a bit more complicated than a buffer.  If a template is
>>> redefined (withdrawn and re-used), data with an old timestamp will
>>> need interpreted using the old template.  I'm not suggesting you
>>> add all the details to the document, just pointing out that
>>> implementing a "buffer" would have some challenges :)
>> 
>> Absolutely. Would you suggest a better word than "buffer" here? (Reading it again, I can see that there might be confusion between an "event buffer" and a uint8_t*…
> 
> "Collecting Processes MAY store older versions of the template for a time in order to handle out-of-order Template management events"?

Hm. it's not just the templates they'll need to store; maybe they should buffer sets, too; again, we don't necessarily want to constrain collector implementors to do things a certain way, we just want to make sure they might get stuff from exporters that it will be very easy to interpret incorrectly...

Cheers,

Brian
_______________________________________________
IPFIX mailing list
[email protected]
https://www.ietf.org/mailman/listinfo/ipfix