Re: [IPFIX] draft-ietf-ipfix-a9n

Hi, Rahul,

Thank you very much for the comments! Comments thereon inline.

On Feb 20, 2012, at 9:07 PM, Rahul Patel wrote:

> Hi,
> 
> Here are my comments.
> 
> I think correlation is important part of aggregation and this draft should
> include a section on it.
> There are different cases of correlation which the draft should touch.

I'm not sure I agree, but it depends on what you mean by correlation. More details below.

> 1. Key fields of Aggregated Flow is same or the subset of all original
> Flows
>     * In this case the correlation is straight forward. The values from
> the original flows are simply merged (added/replaced/etc) into matching
> aggregated flow.

There is a _lot_ of complexity hiding in that "added/replaced/etc". For "added" (that is, the reduced flow key as per section 5.2 of -a9n represents a different Original Flow, i.e., a completely different set of observed packets from the point of view of the sockets at the endpoints) it is so straightforward it's already covered: this isn't really correlation, but spatial aggregation, in the terminology of this document.

For "replaced" this requires the correlation process to know the configuration of the Original Exporters so that it can differentiate Original Flows representing different observed packets from the endpoint POV from Original Flows representing the same packets. This is where correlation gets interesting, and it's my opinion that it's also out of scope for this document.

For "etc" -- here's where it gets interesting, because there might be significant analysis behind trying to figure out what you should do with a set/cluster of N flows that look like each other. And now we're out of scope for the WG, IMO.

> 2. Key fields of Aggregated Flow is larger than at least one original flow.
>     * An example is best to describe this case. One original flow contain
> (5-tuple, src-interface, byte) and another original flow contains
> (5-tuple, transaction-delay). Now the aggregated flow has the key fields
> of srcip, dstip and src-interface). One of the flow doesn't have
> information on per source-interface basis. So the aggregator will perform
> aggregation at two level and represent that data in ipfix-structured
> format. The above data would look like
> 
> Srcip, dstip, transaction_delay, { List of {src-interface, byte } }
> =====  =====  =================  ==================================
> A,     B,     200,               { (Ethernet0/0, 1500),
>                                   (Ethernet1/0, 1600) .. }
> A,     C,     300,               { (Ethernet0/0, 1500) }

Correlating {5-tuple, src-interface, octet-count} and {5-tuple, delay} into {5-tuple, src-interface, octet-count, delay} is a completely different operation than aggregating {5-tuple, src-interface, octet-count, delay} into {src address, delay}, for example, and it really doesn't make any sense to me to try to wedge these into the same framework. Sure, it makes sense for a lot of applications to correlate before aggregating (and the document touches on this already) but that doesn't mean they're the same operation.

I will note that this second example (tweaked; see next para) would be a good basis for a new draft on an intermediate correlation process. But I don't think that draft is this draft.

(As an aside, I find this example scary because it hides a key flow attribute, byte count, behind 6313 structured data, without any particular reason to do so. I am concerned that such overuse of 6313 poses a serious threat to the interoperability and adoption of IPFIX. But that is out of scope for this email.)

So, with respect to the inclusion of these comments in the draft, I am strongly disinclined to do so.  As author, I'll note that we decided that correlation was explicitly out of scope for this draft (paragraph 4 of section 3). As a WG contributor, I'll note that correlation is explicitly not mentioned in the charter item for this draft, and that correlation and aggregation are treated separately in 5982 and 6183. So repurposing this draft at this point would go against what I believe is the sense of the WG to date with respect to the relationship between correlation and aggregation, so I wouldn't be comfortable creeping the scope of the draft without a recharter.

> Section 5.1.1 Distributing values across Interval
> -------------------------------------------------
> * should add 'synchronized expiry of Original Flows' - Under this method
> Flow C would be exported three times with start-time/end-time matching the
> aggregation interval start-time/end-time.

I don't see how the result of that flow is an Aggregated Flow as per section 2.

> * There will always going to some original flows which will arrive late
> and will not be accounted. It will be good have a cumulative counter of
> such late arrivals on per template id (aggregated flow) basis. This
> cumulative count should be periodically exported along with end-time
> (time-of-snapshot). The counter will provide a degree of confidence in the
> data exported by the aggregator for each template.

You can always keep this from happening by lengthening the time you are willing to wait for aggregation for active + idle + export queue delay + 0.5RTT + collector queue delay; in most sane cases the active timeout should dominate this. 

It probably makes sense to note this in section 6 though.

> Section 5.2 Spatial aggregation
> -------------------------------
> * Is it a good idea to add an example of deriving a key in aggregate flow
> from multiple keys or non-keys or combination of Original keys. E.g
> 5-tuple translated to a session-id from metadata table.
> * Along similar line, derivation of aggregate flow key from multiple keys
> from different Original flows. One flow reports 5-tuple and byte-count.
> The other flow reports 5-tuple and the user-id. The aggregated flow has
> the user-id as key and byte-count as collect. I guess this can be
> considered a correlation.

Interesting suggestions, but I didn't (and can't now) see any way to apply them without inventing new Information Elements to hold the combined keys; the idea was to keep the examples as simple and understandable as possible without confusing the issue by adding additional irrelevant detail. However, it does make it difficult to 

Let me think a bit and see if I can come up with a good combined-key example.

Best regards,

Brian

_______________________________________________
IPFIX mailing list
[email protected]
https://www.ietf.org/mailman/listinfo/ipfix