Re: [IPFIX] timestamps, exporters, and other animals (fwd)

Josh, Brian,

Point taken. I created an open issue for the draft.

Regards, Benoit.
>
> Hi Brian;
>
> In summary - yes! I believe you perceive where I'm coming from most 
> accurately. I would be very interested in specific handling of this 
> issue in the next draft.
>
>> Hi, Josh, all,
>>
>> As I read it, one of the wins of MIB Variable Export is that you get 
>> the whole timestamp machinery of IPFIX for free, including absolute 
>> timestamps wherein individual records can have observation timestamps 
>> down to nanosecond precision completely independent of the export 
>> timestamps. Relative timestamps can be expressed either as negative 
>> offsets to the export time (which can be a little tricky to 
>> implement, mind) or as positive offsets to the system initialization 
>> time (itself exported via options)
>>
>> Now, as you'll see elsewhere on the list, the flexible timestamps 
>> need a little work -- the microsecond and nanosecond types in 
>> particular are confusingly defined in a way that has led to 
>> (not-yet-verified) interoperability failures; the discussion about 
>> fixing this in 5101bis/5102bis is elsewhere on the ipfix list.
>>
>> Simply because an exporter has the _tools_ to export accurate 
>> timestamps, of course, does not mean that it can (e.g. due to 
>> implementation constraints) or will (e.g. due to faults or simple 
>> laziness). And system initialization based relative timestamps can 
>> have their own subtle pitfalls as in NetFlow V9 (on which see our 
>> paper in PAM this year, "Peeling Away Timing Error in NetFlow Data", 
>> author copy at 
>> ftp://ftp.tik.ee.ethz.ch/pub/people/martibur/Netflow-Timing-PAM2011.pdf 
>> -- the cyclic error from lost export time precision are V9-specific, 
>> but particularly the drift and delay components of the error there 
>> are universally applicable to any such arrangement, I think)
>>
>> But the protocol here should give you the room to do what you need 
>> with respect to timing at the implementation level... WRT impacts on 
>> documents, would it be helpful to have a more thorough handling of 
>> IPFIX timestamping in the MIB Export draft (i.e., as an introduction 
>> for SNMP people)?
>>
>> Cheers,
>>
>> Brian
>>
>> On Oct 12, 2011, at 9:47 AM, Josh Bailey wrote:
>>
>>>
>>> Hi Juergen;
>>>
>>> I agree with your statement about SNMP not being designed for 
>>> subsecond precision. And I definitely agree that there are things 
>>> you can do to minimise potential delays and that often those delays 
>>> are implementation specific.
>>>
>>> I don't agree though that sysUpTime is when the packet left the 
>>> agent; especially if the agent is running slowly for whatever 
>>> reason, or if the sysUpTime reply happens to be part of a packet 
>>> containing replies for other variables (perhaps from different CPUs 
>>> in a typical distributed CPU router).
>>>
>>> I also don't agree that cycles are usually measured in minutes. In 
>>> my experience cycles are very much shorter on average.
>>>
>>> As you say having a timestamp per counter while costly is ideal in 
>>> some sense. However I am looking to explore different tradeoffs; for 
>>> example, a timestamp followed by a list of OIDs all current at that 
>>> same timestamp.
>>>
>>> I am aware of counter implementations that could do such a 
>>> "snapshot" operation for example.
>>>
>>> Thanks,
>>>
>>>>>
>>>>> Hi Juergen;
>>>>>
>>>>> I assert that polling the sysUpTime does not take care of the core
>>>>> problem, because you cannot know when sysUpTime itself was sampled
>>>>> (eg. you received it after 1.5s because of network delay and because
>>>>> the packet was queued in the control plane kernel because the
>>>>> control plane CPU was busy).
>>>>
>>>> The SNMP agents controls sysUpTime - so sysUpTime is pretty much the
>>>> time the packet left the agent. If you talk about delays in the
>>>> instrumentation, that is the delay to read the register, then you are
>>>> correct. But once again, SNMP polling cycles are usually counted in
>>>> minutes.
>>>>
>>>>> I don't mind so much that the reply was delayed (though I like fast
>>>>> replies!), but I do mind that there can be a very large uncertainty.
>>>>
>>>> Very large is relative to the precision you want. SNMP was not
>>>> designed with subsecond precision in mind.
>>>>
>>>>> There is also the overhead of requesting sysUpTime itself, not to
>>>>> mention when walking a table, you may time slew over the course of
>>>>> walking the table.
>>>>
>>>> For any counter, you need discontinuity detection. The way you walk a
>>>> table has indeed a big impact in terms of data consistency - the
>>>> recommendation here is to use getbulk and to walk the columns of
>>>> interest concurrently.
>>>>
>>>>> I understand definitely re caching hardware counters, etc, and I
>>>>> think that's fine (and necessary in a system with multiple NMSes),
>>>>> and I don't mind that the counters are cached, but I do want to know
>>>>> when they were cached for me to recover maximum temporal
>>>>> information.
>>>>
>>>> SNMP agents typically do not help you with that - caching is rather
>>>> something implementation specific. And yes, operators have complained
>>>> about SNMP counters on some boxes being less "precise" than what the
>>>> CLI shows. I am no way saying SNMP is a model to choose - I just
>>>> wanted to help making it clear what SNMP does and what not.
>>>>
>>>> If you want high accuracy, then every counter reading must be
>>>> timestamped exactly when the counter is read. This of course has a
>>>> price.
>>>>
>>>> /js
>>>>
>>>> -- 
>>>> Juergen Schoenwaelder           Jacobs University Bremen gGmbH
>>>> Phone: +49 421 200 3587         Campus Ring 1, 28759 Bremen, Germany
>>>> Fax:   +49 421 200 3103 <http://www.jacobs-university.de/>
>>>>
>>>>
>>>>
>>>
>>> -- 
>>> Josh Bailey
>>> _______________________________________________
>>> IPFIX mailing list
>>> [email protected]
>>> https://www.ietf.org/mailman/listinfo/ipfix
>>
>>
>>
>
> -- 
> Josh Bailey
> _______________________________________________
> IPFIX mailing list
> [email protected]
> https://www.ietf.org/mailman/listinfo/ipfix
>
>

_______________________________________________
IPFIX mailing list
[email protected]
https://www.ietf.org/mailman/listinfo/ipfix