Re: [IPFIX] review of draft-trammell-ipfix-a9n-03
Brian Trammell <[email protected]>
| Newsgroups | gmane.ietf.ipfix |
|---|---|
| Message-ID | <[email protected]> |
Hi, Paul,
Many, many thanks for your review; though it apparently applies to an older revision of the document, most of these comments still apply. Replies on specific points inline; points receiving no reply were either accepted into the document without comment, or were no longer applicable to the present revision of the document.
These will be incorporated into a new ietf-04 revision of the document to be published shortly.
Again, many thanks, and best regards,
Brian
On Jun 15, 2012, at 11:33 PM, Paul Aitken wrote:
>> 2. Terminology
>>
>> Terms used in this document that are defined in the Terminology
>> section of the IPFIX Protocol [RFC5101] document are to be
>> interpreted as defined there.
>>
>> The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
>> "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
>> document are to be interpreted as described in [RFC2119].
>>
>> In addition, this document defines the following terms
>>
>> Aggregated Flow: A Flow, as defined by [RFC5101], derived from a
>> set of zero or more original Flows within a defined Aggregation
>> Interval. The two primary differences between a Flow and an
>> Aggregated Flow are (1) that the time interval of a Flow is
>> generally derived from information about the timing of the packets
>> comprising the Flow, while the time interval of an Aggregated Flow
>> are generally externally imposed; and (2) that an Aggregated Flow
>> may represent zero packets (i.e., an assertion that no packets
>> were seen for a given Flow Key in a given time interval). Note
>>
>
> That's an oversight in 5101. We should have allowed a Flow to represent zero or more packets.
> It shouldn't be necessary to define an aggregated flow in order to export that nothing was seen.
>
> This should be fixed in 5101bis, and (2) removed above.
Indeed, the strict reading of the definition of Flow in 5101 does not exclude the zero-packet case ("set of packets" may be empty set) but it is strongly implied by the example. I've applied a change to the working version of 5101bis, and removed point 2 here.
>>
>> partially aggregated Flow: A Flow during processing within an
>> Intermediate Aggregation Process; refers to an intermediate data
>> structure during aggregation within the Intermediate Aggregation
>> Process architecture detailed in Section 4.2.
>>
>
> Are "partially aggregated Flows" really necessary?
We use them all over the document to explain what's going in inside an IAP. (Same, kind of, with Contributing Flows...)
>>
>> 3. Use Cases for IPFIX Aggregation
>>
>> Aggregation, as a common data analysis method, has many applications.
>> When used with a regular Aggregation Interval, it generates time
>> series data from a collection of Flows with discrete intervals. Time
>> series data is itself useful for a wide variety of analysis tasks,
>> such as generating input for network anomaly detection systems, or
>> driving visualizations of volume per time for traffic with specific
>>
>
> Fixated on time here. See earlier comments.
>
> Specifically, aggregation does not "generates time series data". It depends what is aggregated.
>
> Example 8.3 is the only place which acknowledges that time is not required. Yet time isn't special, it's just a field like any other.
I have to disagree that "time isn't special": the time interval of a flow is, in essence, a pseudo key field. The specialness of time is really the only thing that makes aggregation hard to specify. In any streaming analysis application -- which pretty much covers anything you can usefully do on a Mediator -- you can't have meaningful data without timing. At the very least, the time window from the startup of the IAP to the present time is an implicit interval.
So yeah, the document is kind of fixated on time. That's intentional. I'd argue that aggregation that doesn't have anything to do with a time interval is the special case, which can be modeled by discarding timestamps (an equivalent operation to the imposition of an infinite interval). However, it wouldn't hurt to be a little more explicit about this in this section, and to more explicitly state that's what's going on in the example "without timing."
>
>> packets --+ +- IPFIX Messages -+
>> | | |
>> V V V
>> +==================+ +====================+ +=============+
>> | Metering Process | | Collecting Process | | File Reader |
>> | | +====================+ +=============+
>> | | | original Flows |
>> | | V V
>> + - - - - - - - - -+======================================+
>> | Intermediate Aggregation Process (IAP) |
>> +=========================================================+
>> | Aggregated Aggregated |
>> | Flows Flows |
>> V V
>> +===================+ +=============+
>> | Exporting Process | | File Writer |
>> +===================+ +=============+
>> | |
>> +------------> IPFIX Messages <----------+
>>
>> Figure 2: Data paths through the aggregation process
>>
>
> An IPFIX File Reader consumes IPFIX Files, not IPFIX Messages.
> An IPFIX File Write produces IPFIX Files, not IPFIX Messages.
>
> Surely the process for (re)generating IPFIX messages is:
>
> IPFIX Messages -> File Writer -> IPFIX File -> File Reader -> Exporting Process -> IPFIX Messages
A File is simply a collection of serialized Messages, though. But good point, the change would make it clearer...
>
>> 4.2. Intermediate Aggregation Process Architecture
>>
>> Within this document, an Intermediate Aggregation Process can be seen
>> as hosting an Intermediate Aggregation Function composed of four
>> types of operations on the intermediate results of aggregation, which
>>
>>
>>
>> Trammell, et al. Expires December 31, 2011 [Page 8]
>> Internet-Draft IPFIX Aggregation June 2011
>>
>>
>> are called partially aggregated Flows in this document, as
>> illustrated in Figure 3.
>>
>> original Flows
>>
>
> Note that here you say "original Flows" rather than contributing Flows. See my earlier comment.
When they show up at the interval distribution step, they're not contributing yet. "Contributing Flows" only has a meaning within the context of a specific Aggregated Flow or set thereof.
>>
>> Interval distribution is a temporal aggregation operation which
>> imposes an Aggregation Interval on the partially aggregated Flow.
>> This Aggregation Interval may be regular, irregular, or derived
>> from the timing of the original Flows themselves. Interval
>> distribution is discussed in detail in Section 5.1.
>>
>
> Must this be done first?
No, see below.
>> Key aggregation is a spatial aggregation operation which results in
>> the addition, modification, or deletion of Flow Key fields in the
>> partially aggregated Flows. New Flow Key fields may be derived
>> from existing Flow Key fields (e.g., looking up an AS number for
>> an IP address), or "promoted" from non-Key fields (e.g., when
>> aggregating Flows by packet count per Flow). Key aggregation can
>>
>
> The above sounds like any non-key can be promoted, which isn't true and needs to be discussed (see later).
>
> Consider, "or derived from certain specific non-Key fields" ?
Okay... although I'm not convinced there exist non-promoteable fields. Certainly some of these might not make a whole lot of sense sense, but you can always count all incoming flows that have some property equal to each distinct value of that property.
>> also add new non-Key fields derived from Key Fields that are
>> deleted during key aggregation; mainly counters of unique reduced
>> keys. Key aggregation is discussed in detail in Section 5.2.
>>
>>
>>
>> Trammell, et al. Expires December 31, 2011 [Page 9]
>> Internet-Draft IPFIX Aggregation June 2011
>>
>>
>> Value aggregation is a spatial aggregation operation which results
>> in the addition, modification, or deletion of non-Key fields in
>> the partially aggregated Flows. These non-Key fields may be
>> "demoted" from existing Key fields, or derived from existing Key
>> or non-Key fields. Value aggregation is discussed in detail in
>> Section 5.3.
>>
>> Aggregate combination combines multiple partially aggregated Flows
>> having undergone interval distribution, key aggregation, and value
>> aggregation which share Flow Keys and Aggregation Intervals into a
>> single aggregated Flow per Flow Key and Aggregation Interval.
>> Aggregate combination is discussed in detail in Section 5.4.
>>
>> The first three of these operations may be carried out any number of
>> times in any order, either on original Flows or on the results of one
>>
>
> "in any order" contradicts Figure 3, which distinctly shows that "interval distribution" must be first.
It is intended to show that at least one interval distribution step is required, as described below. This diagram has undergone significant changes (everyone has had something they hated about it) since the reviewed revision. Representing the associative closure of (interval-distribution[1..x], key-aggregation[0..x], value-aggregation[0..x]) is difficult. Would it help in understanding if the diagram appeared _after_ the normative text?
>> of the Operations (called partially aggregated Flows), with one
>
> "Operations" isn't defined. Either define it, or de-capitalise it.
done (as opposed to Done. :) )
>> caveat. Since Flows carry their own interval data, any spatial
>> aggregation operation implies a temporal aggregation operation, so at
>> least one interval distribution step, even if implicit, is required
>> by this architecture. This is shown as the first step for the sake
>> of simplicity in the diagram above. Once all aggregation operations
>> are complete, aggregate combination ensures that for a given
>> Aggregation Interval, Flow Key, and Observation Domain, only one Flow
>> is produced by the Intermediate Aggregation Process.
>>
>>
>> 5. IP Flow Aggregation Operations
>>
>> As stated in Section 2, an Aggregated Flow is simply an IPFIX Flow
>> generated from original Flows by an Aggregation Function. Here, we
>>
>> detail the operations by which this is achieved within an
>> Intermediate Aggregation Process.
>>
>> 5.1. Temporal Aggregation through Interval Distribution
>>
>> Interval distribution imposes a time interval on the resulting
>> Aggregated Flows. The selection of an interval is specific to the
>> given aggregation application. Intervals may be derived from the
>> original Flows themselves (e.g., an interval may be selected to cover
>> the entire interval containing the set of all Flows sharing a given
>> Key, as in Time Composition describe in Section 5.1.2) or externally
>> imposed; in the latter case the externally imposed interval may be
>> regular (e.g., every five minutes) or irregular (e.g., to allow for
>> different time resolutions at different times of day, under different
>> network conditions, or indeed for different sets of original Flows).
>>
>
> How does the Aggregated Flow consumer know what the aggregation interval was, especially in the "irregular" case?
> ie, with regular intervals, a collector can simply store the received Aggregates as eg hourly reports.
>
> However, with irregular intervals it may have two minutes here and ten minutes there... which may not be easy to process in a meaningful way?
Indeed; for irregular intervals the post-CP processes will either need additional information about how the intervals are distributed, or the analysis will need to be done in a way that is interval-independent. The point is that flows with irregular imposed intervals are still flows, and not to constrain applications which for some reason (I would suspect generally data reduction of "uninteresting" flows through small keys and big intervals) need to use irregular intervals from running on an "IAP" as defined.
>> The length of the imposed interval itself has tradeoffs. Shorter
>> intervals allow higher resolution aggregated data and, in streaming
>>
>>
>>
>> Trammell, et al. Expires December 31, 2011 [Page 10]
>> Internet-Draft IPFIX Aggregation June 2011
>>
>>
>> applications, faster reaction time. Longer intervals lead to greater
>> data reduction and simplified counter distribution. Specifically,
>>
>
> *may* lead to greater data reduction, though not necessarily.
True.
>> counter distribution is greatly simplified by the choice of an
>> interval longer than the duration of longest original Flow, itself
>> generally determined by the original Flow's Metering Process active
>> timeout; in this case an original Flow can contribute to at most two
>> Aggregated Flows, and the more complex value distribution methods
>> become inapplicable.
>>
>
> This only considers the Metering Process; it fails to consider any delay in the Exporting Process.
>
> eg, consider and implementation with 64K cache entries which are checked for exporting at a rate of 1000 entries/second. It will take > 1 minute to check and export the entire cache.
And the timestamps are put on the flows by the EP as opposed to the MP?
> Again, consider an implementation which aims to maximise the size of each export packet, building it up a record at a time as records expire from the cache, until the export packet is quite full. If records expire slowly then the mechanism may introduce a significant delay.
Again, this will affect the _record_ timestamps? If this is the case, then the guidance would be to choose intervals larger than the active timeout plus the MP-EP latency, but this seems to be limited to specific implementations which separate timing in this way.
> In effect both of these delays cause flows to appear in later intervals than may be expected. eg, in Figure 4, "Flow A" may appear in interval 1 or interval 2, rather than interval 0.
>
>
>> | | | |
>> | |<--Flow A-->| | | |
>> | |<--Flow B-->| | |
>> | |<-------------Flow C-------------->| |
>> | | | |
>> | interval 0 | interval 1 | interval 2 |
>>
>> Figure 4: Illustration of interval distribution
<snip>
>
>> from correlation of the original Flow information with some external
>> source. There are two basic operations here. First, Aggregated Flow
>> Keys may be derived directly from original Flow Keys through
>> reduction, or the dropping of fields or precision in the original
>> Flow Keys. Second, an Aggregated Flow Key may be derived through
>> replacement, e.g. by removing one or more fields from the original
>> Flow and replacing them with a fields derived from the removed
>> fields. Replacement may refer to external information (e.g., IP to
>> AS number mappings). Replacement need not replace only key fields.
>> For example, consider an application which aggregates flows by packet
>> count (i.e., generating an Aggregated Flow for all one-packet Flows,
>> one for all two-packet Flows, and so on). This application would
>> promote the packet count to a Flow Key field.
>>
>
> Say more about promotion, ie it can only be done on non-key fields which represent a single unique value. eg, counters, time.
> If a NK field might be any value from a set of >1 values (eg, an IP address, port number, AS, TOS) then it's not suitable for promotion to Key, because there's no way of knowing which other values were or were not observed, and therefore no way of dividing the flow amongst those values.
I'm not sure I follow you here. Why does the potential set of values (and its difference to the actual set) affect the promotion ability? Can you give a more detailed example?
>>
>> Trammell, et al. Expires December 31, 2011 [Page 13]
>> Internet-Draft IPFIX Aggregation June 2011
>>
>>
>> Key aggregation may also result in the addition of new non-Key fields
>> to the Aggregated Flows, namely original Flow counters and unique
>> reduced key counters; these are treated in more detail in
>> Section 5.2.2 and Section 5.2.1, respectively.
>>
>
> I'm sure other non-keys could be derived too, though a good example doesn't immediately spring to mind.
They're also not handled in the draft; we originally thought about going further down this road but decided to keep the list short and general in the interests of (1) brevity and (2) IE number space conservation.
>> In any key aggregation operation, reduction and/or replacement may be
>> applied any number of times in any order. Which of these operations
>> are supported by a given implementation is implementation- and
>> application-dependent. Key aggregation may aggregate original Flows
>> with different sets of Flow Key fields; only the Flow Keys of the
>> resulting Aggregated Flows of any given Key aggregation operation
>> need contain the same set of fields.
>>
>
> I don't understand what "only ... fields" is saying.
The Key Fields defining the input to the operation can be heterogeneous, because it's only the outputs that need to match; since these operations can be arbitrarily split and combined, it's only the final key fields that need to match. This is a confusing sentence, though, and adds little, so I've decided to cut it.
>> 5.4. Aggregation Combination
>>
>> Interval distribution and key aggregation together may generate
>> multiple partially aggregated Flows covering the same time interval
>> with the same Flow Key. The process of combining these partially
>> aggregated Flows into a single Aggregated Flow is called aggregation
>> combination. In general, non-Key values from multiple contributing
>> Flows are combined using the same operation by which values are
>> combined from packets to form Flows for each Information Element.
>> Counters are summed, averages are averaged, flags are unioned, and so
>> on.
>>
>
> Although this is common knowledge for us, is it actually specified anywhere in IPFIX?
>
> ie, should [IANA-IPFIX] list the relevant aggregation operation for each field? eg, min, max, sum, union, none?
We thought about this, long ago; however, (1) there may be some applications wherein non-natural operations on aggregated values would be useful and (2) specifying in the registry that only one operation was permissible gets us close to specifying implementation details of the IAP, which we want to avoid...
>> 7. Export of Aggregated IP Flows using IPFIX
>>
>> In general, Aggregated Flows are exported in IPFIX as any normal
>> Flow. However, certain aspects of Aggregated Flow export benefit
>> from additional guidelines, or new Information Elements to represent
>>
>>
>>
>> Trammell, et al. Expires December 31, 2011 [Page 17]
>> Internet-Draft IPFIX Aggregation June 2011
>>
>>
>> aggregation metadata or information generated during aggregation.
>> These are detailed in the following subsections.
>>
>
> Have the new IEs been implemented in released code?
> We should avoid defining theoretical new IEs (ie, those which nobody has actually implemented, or plans to implement).
> "Rough consensus and _running code_" :-)
These fields are often used as output of rwuniq, for one; when I left CERT (admittedly, a long time ago now) there were plans IIRC to introduce IPFIX-based intermediate output for the SiLK analysis tools. I don't know whether these are still current.
>>
>> 7.2.4. originalFlows
>>
>
> For consistency with the above IEs, name this "originalFlowsContributing" ?
It was pointed out that this is probably covered by deltaFlowCount (IE 3); the -ietf rev of the document was changed accordingly. (However, float64 is not the correct type; have corrected this...
>> Description: The conservative count of original Flows contributing
>> to this Aggregated Flow; may be distributed via any of the methods
>> described in Section 5.1.1.
>>
>> Abstract Data Type: float64
>>
>> ElementId: 3
>>
>> Status: Current
>>
>>
>>
>> 7.4.1. Aggregate Counter Distribution Options Template
>>
>> This Options Template defines the Aggregate Counter Distribution
>> Record, which allows the binding of a value distribution method to a
>> Template ID. This is used to signal to the Collecting Process how
>> the counters were distributed. The fields are as below:
>>
>> +-------------------------+-----------------------------------------+
>> | IE | Description |
>> +-------------------------+-----------------------------------------+
>> | templateId [scope] | The Template ID of the Template |
>> | | defining the Aggregated Flows to which |
>> | | this distribution option applies. This |
>> | | Information Element MUST be defined as |
>> | | a Scope Field. |
>> | valueDistributionMethod | The method used to distribute the |
>> | | counters for the Aggregated Flows |
>> | | defined by the associated Template. |
>> +-------------------------+-----------------------------------------+
>>
>
> Presumably the scope could be more specific, eg an "informationElementIndex" could be included.
Per-IE indexing would be done as in 5610, presumably, but it's not clear that it makes sense to distribute the values of separate fields separately; see the next paragraph.
>
>> 7.4.2. valueDistributionMethod Information Element
>>
>> Description: A description of the method used to distribute the
>> counters from contributing Flows into the Aggregated Flow records
>> described by an associated Template. The method is deemed to
>> apply to all the non-key Information Elements in the referenced
>> Template for which value distribution is a valid operation; if the
>>
>
> No; it applies to the specified scope, which happens to be a templateId in this case - but may be something else in other uses.
Point. However, it is not at _all_ clear that it makes any sense to distribute one counter one way and another counter another for a given Aggregated Flow template: when aggregating to Aggregated Flows with uniform counter distribution, each Aggregated Flow remains independent as a Flow; otherwise, the resulting Aggregated Flows are dependent among the set of flows to which a given value may be distributed.
> eg, vDM could be used with structured data, where the intention would be that it applies only to the current relevant structure, and not to all the non-key elements in some other template.
>
>> originalFlowsInitiated and/or originalFlowsCompleted Information
>> Elements appear in the Template, they are not subject to this
>> distribution method, as they each infer their own distribution
>> method. The distribution methods are taken from Section 5.1.1 and
>> encoded as follows:
>>
>
> Please separate the table entries below to make them easier to read.
>
>
>> +-------+-----------------------------------------------------------+
>> | Value | Description |
>> +-------+-----------------------------------------------------------+
>>
>
> What does value = zero mean?
Missing; will add a "less-than-default" meaning for this (as in "explictly not distributed according to any consistent policy")
>
>> | 1 | Start Interval: The counters for an original Flow are |
>> | | added to the counters of the appropriate Aggregated Flow |
>> | | containing the start time of the original Flow. This |
>> | | should be assumed the default if value distribution |
>> | | information is not available at a Collecting Process for |
>> | | an Aggregated Flow. |
>> | 2 | End Interval: The counters for an original Flow are added |
>> | | to the counters of the appropriate Aggregated Flow |
>> | | containing the end time of the original Flow. |
>>
>>
>>
>>
>>
>>
>> Trammell, et al. Expires December 31, 2011 [Page 21]
>> Internet-Draft IPFIX Aggregation June 2011
>>
>>
>> | 3 | Mid Interval: The counters for an original Flow are added |
>> | | to the counters of a single appropriate Aggregated Flow |
>> | | containing some timestamp between start and end time of |
>> | | the original Flow. |
>> | 4 | Simple Uniform Distribution: Each counter for an original |
>> | | Flow is divided by the number of time intervals the |
>> | | original Flow covers (i.e., of appropriate Aggregated |
>> | | Flows sharing the same Flow Key), and this number is |
>> | | added to each corresponding counter in each Aggregated |
>> | | Flow. |
>> | 5 | Proportional Uniform Distribution: Each counter for an |
>> | | original Flow is divided by the number of time _units_ |
>> | | the original Flow covers, to derive a mean count rate. |
>> | | This mean count rate is then multiplied by the number of |
>> | | time units in the intersection of the duration of the |
>> | | original Flow and the time interval of each Aggregated |
>> | | Flow. This is like simple uniform distribution, but |
>> | | accounts for the fractional portions of a time interval |
>> | | covered by an original Flow in the first and last time |
>> | | interval. |
>> | 6 | Simulated Process: Each counter of the original Flow is |
>> | | distributed among the intervals of the Aggregated Flows |
>> | | according to some function the Aggregation Process uses |
>> | | based upon properties of Flows presumed to be like the |
>> | | original Flow. This is essentially an assertion that the |
>> | | Aggregation Process has no direct packet timing |
>> | | information but is nevertheless not using one of the |
>> | | other simpler distribution methods. The Aggregation |
>> | | Process specifically makes no assertion as to the |
>> | | correctness of the simulation. |
>> | 7 | Direct: The Aggregation Process has access to the |
>> | | original packet timings from the packets making up the |
>> | | original Flow, and uses these to distribute or |
>> | | recalculate the counters. |
>> +-------+-----------------------------------------------------------+
>>
>
> State whether this can be extended in future, or whether a new IE would be required.
>
> From an IE-doctors point of view, this should generally be stated for all list-based IEs, since someone could extend the list and claim support for this IE, while someone else claims support with the original, more limited, list. Clearly the two "supporting" processes would not be interoperable.
>
> Expressed another way: if the list is extensible then a new implementation which extends the list invalidates all existing implementations.
>
> However, requiring new IEs for each new value may be unreasonable, leading to fragmentation (values 1-7 in IE#1, 8-9 in IE#2, 10-12 in IE#3) and unnecessarily hastening IE space exhaustion.
The correct way to handle this would be to create a subregistry; the set is intended to be complete, so we didn't deem a subregistry to be necessary.
>> The data records given as input to the examples in this section are
>> shown below, in the format "flowStartMilliseconds-flowEndMilliseconds
>> sourceIPv4Address:sourceTransportPort -> destinationIPv4Address:
>> destinationTransportPort (protocolIdentifier) octetDeltaCount";
>> timestamps are given in H:MM:SS.sss format.
>>
>> 9:00:00.138-9:00:00.138 192.0.2.2:47113 -> 192.0.2.131:53 (17) 119
>> 9:00:03.246-9:00:03.246 192.0.2.2:22153 -> 192.0.2.131:53 (17) 83
>> 9:00:00.478-9:00:03.486 192.0.2.2:52420 -> 198.51.100.2:443 (6) 1637
>> 9:00:07.172-9:00:07.172 192.0.2.3:56047 -> 192.0.2.131:53 (17) 111
>> 9:00:07.309-9:00:14.861 192.0.2.3:41183 -> 198.51.100.67:80 (6) 16838
>> 9:00:03.556-9:00:19.876 192.0.2.2:17606 -> 198.51.100.68:80 (6) 11538
>> 9:00:25.210-9:00:25.210 192.0.2.3:47113 -> 192.0.2.131:53 (17) 119
>> 9:00:26.358-9:00:30.198 192.0.2.3:48458 -> 198.51.100.133:80 (6) 2973
>> 9:00:29.213-9:01:00.061 192.0.2.4:61295 -> 198.51.100.2:443 (6) 8350
>> 9:04:00.207-9:04:04.431 203.0.113.3:41256 -> 198.51.100.133:80 (6) 778
>> 9:03:59.624-9:04:06.984 203.0.113.3:51662 -> 198.51.100.3:80 (6) 883
>> 9:06:56.813-9:06:59.821 203.0.113.3:52572 -> 198.51.100.2:443 (6) 1637
>> 9:06:30.565-9:07:00.261 203.0.113.3:49914 -> 197.51.100.133:80 (6) 561
>> 9:06:55.160-9:07:05.208 192.0.2.2:50824 -> 198.51.100.2:443 (6) 1899
>> 9:06:49.322-9:07:05.322 192.0.2.3:34597 -> 198.51.100.3:80 (6) 1284
>> 9:07:05.849-9:07:09.625 203.0.113.3:58907 -> 198.51.100.4:80 (6) 2670
>> 9:10:45.161-9:10:45.161 192.0.2.4:22478 -> 192.0.2.131:53 (17) 75
>> 9:10:45.209-9:11:01.465 192.0.2.4:49513 -> 198.51.100.68:80 (6) 3374
>> 9:10:57.094-9:11:00.614 192.0.2.4:64832 -> 198.51.100.67:80 (6) 138
>> 9:10:59.770-9:11:02.842 192.0.2.3:60833 -> 198.51.100.69:443 (6) 2325
>> 9:13:53.933-9:14:06.605 192.0.2.2:19638 -> 198.51.100.3:80 (6) 2869
>> 9:13:02.864-9:14:08.720 192.0.2.3:40429 -> 198.51.100.4:80 (6) 18289
>>
>> Figure 8: Input data for examples
>>
>
> Consider listing the values in aligned columns without the syntax?
> It fits into 76 chars with double-spaced columns, and is slightly easier to read.
> Also, good to add column titles to make the subsequent figures easier to follow.
Done.
> Finally, the data are a little out of order. Although it really doesn't matter, in-order might make the example a little easier to follow for pedants who really want to check your data manually, like me.
The out-of-order aspect is intentional; it reflects a realistic export ordering for the input data. However, this can be handled in the prose...
>> 9:00:00.000-9:05:00.000 192.0.2.2:47113 -> 192.0.2.131:53 (17) 119
>> 9:00:00.000-9:05:00.000 192.0.2.2:22153 -> 192.0.2.131:53 (17) 83
>> 9:00:00.000-9:05:00.000 192.0.2.2:52420 -> 198.51.100.2:443 (6) 1637
>> 9:00:00.000-9:05:00.000 192.0.2.3:56047 -> 192.0.2.131:53 (17) 111
>> 9:00:00.000-9:05:00.000 192.0.2.3:41183 -> 198.51.100.67:80 (6) 16838
>> 9:00:00.000-9:05:00.000 192.0.2.2:17606 -> 198.51.100.68:80 (6) 11538
>> 9:00:00.000-9:05:00.000 192.0.2.3:47113 -> 192.0.2.131:53 (17) 119
>> 9:00:00.000-9:05:00.000 192.0.2.3:48458 -> 198.51.100.133:80 (6) 2973
>> 9:00:00.000-9:05:00.000 192.0.2.4:61295 -> 198.51.100.2:443 (6) 8350
>> 9:00:00.000-9:05:00.000 203.0.113.3:41256 -> 198.51.100.133:80 (6) 778
>> 9:00:00.000-9:05:00.000 203.0.113.3:51662 -> 198.51.100.3:80 (6) 883
>> 9:05:00.000-9:10:00.000 203.0.113.3:52572 -> 198.51.100.2:443 (6) 1637
>> 9:05:00.000-9:10:00.000 203.0.113.3:49914 -> 197.51.100.133:80 (6) 561
>> 9:05:00.000-9:10:00.000 192.0.2.2:50824 -> 198.51.100.2:443 (6) 1899
>> 9:05:00.000-9:10:00.000 192.0.2.3:34597 -> 198.51.100.3:80 (6) 1284
>> 9:05:00.000-9:10:00.000 203.0.113.3:58907 -> 198.51.100.4:80 (6) 2670
>> 9:10:00.000-9:15:00.000 192.0.2.4:22478 -> 192.0.2.131:53 (17) 75
>> 9:10:00.000-9:15:00.000 192.0.2.4:49513 -> 198.51.100.68:80 (6) 3374
>> 9:10:00.000-9:15:00.000 192.0.2.4:64832 -> 198.51.100.67:80 (6) 138
>> 9:10:00.000-9:15:00.000 192.0.2.3:60833 -> 198.51.100.69:443 (6) 2325
>> 9:10:00.000-9:15:00.000 192.0.2.2:19638 -> 198.51.100.3:80 (6) 2869
>> 9:10:00.000-9:15:00.000 192.0.2.3:40429 -> 198.51.100.4:80 (6) 18289
>>
>> Figure 11: Partially aggregated Flows: intervals imposed
>>
>
> Why two timestamps? It's unclear which interval a flow at 9:05:00.000 should be in.
> Whereas listing a single timestamp (9:00, 9:05, 9:10) is less ambiguous.
Notionally, each flow contains the interval to which it is binned.
>> 9:00:00.000-9:05:00.000 192.0.2.2 13377
>> 9:00:00.000-9:05:00.000 192.0.2.3 20041
>> 9:00:00.000-9:05:00.000 192.0.2.4 8350
>> 9:00:00.000-9:05:00.000 203.0.113.3 1661
>> 9:05:00.000-9:10:00.000 192.0.2.2 1899
>> 9:05:00.000-9:10:00.000 192.0.2.3 1284
>> 9:05:00.000-9:10:00.000 203.0.113.3 4868
>> 9:10:00.000-9:15:00.000 192.0.2.2 2869
>> 9:10:00.000-9:15:00.000 192.0.2.3 20594
>>
>
> 20594 should be 2325 + 18289 = 20614.
>
>
>> 9:10:00.000-9:15:00.000 192.0.2.4 3587
>>
>> Figure 14: Aggregated Flows
>>
>
> As a cross-check, sum(Figure_8) = sum(Figure_11) = 78550, while sum(Figure_14) = 78530 - so there are 20 octets missing.
>
Thanks for checking the math here... Should have thrown these into a calculator but I must admit I seem to recall doing the math in my head for these...
>> 8.3. Distinct Source Count per Destination Endpoint
>>
>> Aggregating flows by destination address and port, and counting
>> distinct sources aggregated away, can be used as part of passive
>> service inventory and host characterization approaches. This example
>> shows aggregation as an analysis technique, performed on source data
>> stored in an IPFIX File. As the Transport Session in this File is
>> bounded, removal of all timestamp information allows summarization of
>> the entire time interval contained within the interval. Removal of
>> timing information during interval imposition is equivalent to an
>> infinitely long imposed time interval. This demonstrates both how
>> infinite intervals work, and how unique counters work.
>>
>
> At last, an example which shows that time is _not_ the principal factor in every aggregation - and is not even required.
>
> Since there's no temporal aggregation here, there's no interval distribution - which contradicts figure 3.
Ah, but there is temporal aggregation - of all input data to the process is accounted to the same interval.
_______________________________________________
IPFIX mailing list
[email protected]
https://www.ietf.org/mailman/listinfo/ipfix