Re: [IPFIX] draft-ietf-ipfix-a9n

Hi Brian,

Comments inline..[RP]

On 2/21/12 5:23 AM, "Brian Trammell" <[email protected]> wrote:

>Hi, Rahul,
>
>Thank you very much for the comments! Comments thereon inline.
>
>On Feb 20, 2012, at 9:07 PM, Rahul Patel wrote:
>
>> Hi,
>> 
>> Here are my comments.
>> 
>> I think correlation is important part of aggregation and this draft
>>should
>> include a section on it.
>> There are different cases of correlation which the draft should touch.
>
>I'm not sure I agree, but it depends on what you mean by correlation.
>More details below.
>
>> 1. Key fields of Aggregated Flow is same or the subset of all original
>> Flows
>>     * In this case the correlation is straight forward. The values from
>> the original flows are simply merged (added/replaced/etc) into matching
>> aggregated flow.
>
>There is a _lot_ of complexity hiding in that "added/replaced/etc". For
>"added" (that is, the reduced flow key as per section 5.2 of -a9n
>represents a different Original Flow, i.e., a completely different set of
>observed packets from the point of view of the sockets at the endpoints)
>it is so straightforward it's already covered: this isn't really
>correlation, but spatial aggregation, in the terminology of this document.
>
>For "replaced" this requires the correlation process to know the
>configuration of the Original Exporters so that it can differentiate
>Original Flows representing different observed packets from the endpoint
>POV from Original Flows representing the same packets. This is where
>correlation gets interesting, and it's my opinion that it's also out of
>scope for this document.
>
>For "etc" -- here's where it gets interesting, because there might be
>significant analysis behind trying to figure out what you should do with
>a set/cluster of N flows that look like each other. And now we're out of
>scope for the WG, IMO.

[RP] 'added/replaced/etc' referred to non-key fields operation during
aggregation. Counters will be added, min and max values may be replaced,
averages would be recalculated, etc.

>
>> 2. Key fields of Aggregated Flow is larger than at least one original
>>flow.
>>     * An example is best to describe this case. One original flow
>>contain
>> (5-tuple, src-interface, byte) and another original flow contains
>> (5-tuple, transaction-delay). Now the aggregated flow has the key fields
>> of srcip, dstip and src-interface). One of the flow doesn't have
>> information on per source-interface basis. So the aggregator will
>>perform
>> aggregation at two level and represent that data in ipfix-structured
>> format. The above data would look like
>> 
>> Srcip, dstip, transaction_delay, { List of {src-interface, byte } }
>> =====  =====  =================  ==================================
>> A,     B,     200,               { (Ethernet0/0, 1500),
>>                                   (Ethernet1/0, 1600) .. }
>> A,     C,     300,               { (Ethernet0/0, 1500) }
>
>Correlating {5-tuple, src-interface, octet-count} and {5-tuple, delay}
>into {5-tuple, src-interface, octet-count, delay} is a completely
>different operation than aggregating {5-tuple, src-interface,
>octet-count, delay} into {src address, delay}, for example, and it really
>doesn't make any sense to me to try to wedge these into the same
>framework. Sure, it makes sense for a lot of applications to correlate
>before aggregating (and the document touches on this already) but that
>doesn't mean they're the same operation.
>
>I will note that this second example (tweaked; see next para) would be a
>good basis for a new draft on an intermediate correlation process. But I
>don't think that draft is this draft.
>
>(As an aside, I find this example scary because it hides a key flow
>attribute, byte count, behind 6313 structured data, without any
>particular reason to do so. I am concerned that such overuse of 6313
>poses a serious threat to the interoperability and adoption of IPFIX. But
>that is out of scope for this email.)
>
>So, with respect to the inclusion of these comments in the draft, I am
>strongly disinclined to do so.  As author, I'll note that we decided that
>correlation was explicitly out of scope for this draft (paragraph 4 of
>section 3). As a WG contributor, I'll note that correlation is explicitly
>not mentioned in the charter item for this draft, and that correlation
>and aggregation are treated separately in 5982 and 6183. So repurposing
>this draft at this point would go against what I believe is the sense of
>the WG to date with respect to the relationship between correlation and
>aggregation, so I wouldn't be comfortable creeping the scope of the draft
>without a recharter.

[RP] I am fine with a different draft and reference it in this draft.
 
>
>> Section 5.1.1 Distributing values across Interval
>> -------------------------------------------------
>> * should add 'synchronized expiry of Original Flows' - Under this method
>> Flow C would be exported three times with start-time/end-time matching
>>the
>> aggregation interval start-time/end-time.
>
>I don't see how the result of that flow is an Aggregated Flow as per
>section 2.

[RP] It is an aggregated flow. It is very similar to Proportional Uniform
Distribution (section 5.1.1) but it is more accurate

   |                |                |                |
   | |<--Flow A-->| |                |                |
   |       10       |                |                |
   |                |                |                |
   |        |<--Flow B-->|           |                |
   |            6   | 2              |                |  <== Bytes in each
interval
   |                |                |                |
   |          |<-------------Flow C-------------->|   |
   |             4  |    32          |   10           |  <== Bytes in each
interval
   |                |                |                |
   |   interval 0   |   interval 1   |   interval 2   |



Using Synchronized method
=========================
Aggregated Flow will have following bytes
Interval 0 = (10 + 6 + 4) = 20
Interval 1 = (0 + 2 + 32) = 34
Interval 2 = (0 + 0 + 10) = 10


Using Proportional Uniform Distribution,
========================================
(assuming Interval is 30 seconds
 Flow B overlapped 15 secs of interval 0, 7.5 secs interval 1)
 Flow C overlapped 10 secs of interval 0, 30 secs of interval 1, 20 secs
interval 2)

Aggregate Flow will have following bytes
Interval 0 = (10 + (6+2)*(15/22.5) + 46*(10/60)) = 10+5.33+7.66 = 23
Interval 1 = (0 + (6+2)*(7.5/22.5) + 46*(30/60)) = 0+2.66+23    = 25.66

Interval 0 = (0 + 0 + 46*(20/60))                = 0+0+15.33    = 15.33




>
>> * There will always going to some original flows which will arrive late
>> and will not be accounted. It will be good have a cumulative counter of
>> such late arrivals on per template id (aggregated flow) basis. This
>> cumulative count should be periodically exported along with end-time
>> (time-of-snapshot). The counter will provide a degree of confidence in
>>the
>> data exported by the aggregator for each template.
>
>You can always keep this from happening by lengthening the time you are
>willing to wait for aggregation for active + idle + export queue delay +
>0.5RTT + collector queue delay; in most sane cases the active timeout
>should dominate this.
>
>It probably makes sense to note this in section 6 though.

[RP] Some device may not want hold the data for longer to avoid larger
memory foot print. In those case it may be desired to have such counters.
I guess one could say that it is implementation detail.

Thanks,

-Rahul

>
>> Section 5.2 Spatial aggregation
>> -------------------------------
>> * Is it a good idea to add an example of deriving a key in aggregate
>>flow
>> from multiple keys or non-keys or combination of Original keys. E.g
>> 5-tuple translated to a session-id from metadata table.
>> * Along similar line, derivation of aggregate flow key from multiple
>>keys
>> from different Original flows. One flow reports 5-tuple and byte-count.
>> The other flow reports 5-tuple and the user-id. The aggregated flow has
>> the user-id as key and byte-count as collect. I guess this can be
>> considered a correlation.
>
>Interesting suggestions, but I didn't (and can't now) see any way to
>apply them without inventing new Information Elements to hold the
>combined keys; the idea was to keep the examples as simple and
>understandable as possible without confusing the issue by adding
>additional irrelevant detail. However, it does make it difficult to
>
>Let me think a bit and see if I can come up with a good combined-key
>example.
>
>Best regards,
>
>Brian
>


_______________________________________________
IPFIX mailing list
[email protected]
https://www.ietf.org/mailman/listinfo/ipfix