Re: [IPFIX] NetFlow v9 to IPFIX conversion
Paul Aitken <[email protected]> Thu, 16 Apr 2015 10:54:30 +0100
| Newsgroups | gmane.ietf.ipfix |
|---|---|
| Message-ID | <[email protected]> |
--===============3640271284766209822== Content-Type: multipart/alternative; boundary="------------040401040503000409030408" --------------040401040503000409030408 Content-Type: text/plain; charset="utf-8"; format=flowed Content-Transfer-Encoding: 7bit Petr, > the problem we are facing here is that the NetFlow v9 ID space is not > centrally managed NetFlow v9 is a cisco protocol, and the IDs are allocated by Cisco's netflow police. Contact [email protected]. Anyone using NFv9 IDs without contacting them is asking for trouble (ie, ID collisions). > therefore we cannot hope to cover all possible IDs and vendors, since > they can change. Moreover, as you say, the owner of a PEN maintains > his ID registry, therefore we should not use someones PEN just because > he defined the nf9 elements. Equally, nobody should be self-defining NFv9 elements without reference to Cisco, because they've no idea what IDs Cisco (or others) might use in future. > What I propose is similar to RFC 5612. We should define an NetFlow to > IPFIX conversion PEN which could be used by anyone for mapping nf9 > elements with ID > 2^15. The ID of that element would not be changed, > only the PEN would be added, which, as an added benefit, would be > fairly easy to implement in IPFIX mediators. No central management is > necessary, just as you need to know the nf9 elements that you use, you > would have to know the semantics from the appropriate vendor. The > collisions cannot be avoided, however, they are not managed in nf9 either. > > As you say, there is currently no PEN for general experimental use > that could be used as a fallback. Even if there was, you shouldn't use such a PEN in a released product because it could conflict with someone else's usage. P. > I think that creating a specific PEN for this task is the right solution. > > Best regards, > Petr > > On Thu, Apr 2, 2015 at 10:36 PM, Gerhard Muenz <[email protected] > <mailto:[email protected]>> wrote: > > > Hi Petr, > > I do not understand the use case of the new PEN that you suggest. > > It does not make sense to register a PEN for an ID space that is > not centrally managed in a kind of registry because uniqueness of > ID usage must be ensured. Usually, the owner of a PEN maintains > such a registry. > > Is there a registry that lists all vendor specific Netflow 9 Field > Types? > If not, how can you be sure that there are no collisions in the ID > space? > > I think that you need to find and use the PENs of the vendors that > have defined the additional Netflow 9 Field Types IEs, regardless > of whether the IDs are below or above 2^15. Unfortunately, there > is no PEN for general experimental use (at least I have not found > any) that you could use as a fallback if you do not find an > appropriate vendor PEN. > > If you want to use your own non-standard IEs, then you should use > the PEN of your organization: > > 8057 > CESNET > CESNET masters team > masters&cesnet.cz <http://cesnet.cz> > > Maybe, you can also use this PEN to map Field Types for which you > do not find a vendor. > > Regards, > Gerhard > > > > On 26.03.2015 08:14, Petr Velan wrote: >> Hi Andrew, all, >> >> thank you for your explanation regarding nprobe. >> >> However, we also need a fallback for unknown exporters with IEs > >> 2^15. The generic requests for PENs need organization name, >> contact name and email address. I can try to request the PEN for >> NetFlow v9 compatibility myself, but I'd like it to be more >> public. Therefore, I suggest to complete the request with >> something like: >> *Organization Name*: NetFlow v9 to IPFIX >> *Contact Name*: IPFIX WG >> *Contact E-Mail: *[email protected] <mailto:[email protected]> >> >> This is just a first proposal to get things moving, please add >> your thoughts. Once the PEN is granted, we can move forward and >> explain its purpose in a short RFC. >> >> Petr >> >> On Tue, Jan 6, 2015 at 9:07 PM, Andrew Feren <[email protected] >> <mailto:[email protected]>> wrote: >> >> Hi Petr, >> >> On 01/06/2015 07:03 AM, Petr Velan wrote: >>> Hello all, >>> >>> I'm not sure whether this is the right place to ask, but we >>> encountered following problem when converting NetFlow v9 >>> messages to IPFIX. >>> >>> Some vendors (I've heard of ntop) are using elements IDs >>> large than 32767 in NetFlow v9. When converting messages >>> with these elements to IPFIX, they are considered to be >>> Enterprise Numbers. To generate proper IPFIX message, we >>> need to do one of the following: >>> a) Generate a list of the elements and map them to PEN of >>> the correct vendor. However, this would result in an attempt >>> to cover all possible elements that anybody used in NetFlow >>> v9. Moreover, we would still have to somehow handle the >>> cases where the element is unknown >> This should help with ntop/nprobe >> >> Recent versions of nprobe (since version 5.5.5 I think) all >> use the following mapping. >> >> PEN = 35632 and IPFIXID = (v9ID - 57472) >> >> For example, one v9 IE that nprobe exports is >> MYSQL_SERVER_VERSION 57667. The IPFIX equivalent would be >> MYSQL_SERVER_VERSION(35632/195). >> >> The nprobe docs have a complete list. >> >> Older versions of nprobe (pre ~2010) use IEs not in RFC 3954, >> but later allocated in IANA. There is no good way to convert >> those v9 exports to IPFIX. >> >> -Andrew >> >> >>> b) Request a PEN for NetFlow compatibility and just add this >>> PEN for every element that has ID larger than 32767. >>> >>> Personally, I believe that the b) is more general and >>> error-prone. Do you think, that it would be possible to >>> dedicate whole PEN to this cause? >>> >>> Thank you for any opinions, >>> >>> Petr Velan >>> >>> >>> >>> _______________________________________________ >>> IPFIX mailing list >>> [email protected] <mailto:[email protected]> >>> https://www.ietf.org/mailman/listinfo/ipfix >> >> >> >> >> _______________________________________________ >> IPFIX mailing list >> [email protected] <mailto:[email protected]> >> https://www.ietf.org/mailman/listinfo/ipfix > > --------------040401040503000409030408 Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: quoted-printable <html> <head> <meta content=3D"text/html; charset=3Dutf-8" http-equiv=3D"Content-Ty= pe"> </head> <body bgcolor=3D"#FFFFFF" text=3D"#000000"> Petr,<br> <br> <blockquote cite=3D"mid:[email protected]= ail.com" type=3D"cite"> <div dir=3D"ltr"> <div> <div> <div>the problem we are facing here is that the NetFlow v9 ID space is not centrally managed</div> </div> </div> </div> </blockquote> <br> NetFlow v9 is a cisco protocol, and the IDs are allocated by Cisco's netflow police. Contact <a class=3D"moz-txt-link-abbreviated" href=3D= "mailto:[email protected]">[email protected]</a>.<br> <br> Anyone using NFv9 IDs without contacting them is asking for trouble (ie, ID collisions).<br> <br> <br> <blockquote cite=3D"mid:[email protected]= ail.com" type=3D"cite"> <div dir=3D"ltr"> <div> <div> <div>therefore we cannot hope to cover all possible IDs and vendors, since they can change. Moreover, as you say, the owner of a PEN maintains his ID registry, therefore we should not use someones PEN just because he defined the nf9 elements.<br> </div> </div> </div> </div> </blockquote> <br> Equally, nobody should be self-defining NFv9 elements without reference to Cisco, because they've no idea what IDs Cisco (or others) might use in future.<br> <br> <br> <blockquote cite=3D"mid:[email protected]= ail.com" type=3D"cite"> <div dir=3D"ltr"> <div> <div>What I propose is similar to RFC 5612. We should define an NetFlow to IPFIX conversion PEN which could be used by anyone for mapping nf9 elements with ID > 2^15. The ID of that element would not be changed, only the PEN would be added, which, as an added benefit, would be fairly easy to implement in IPFIX mediators. No central management is necessary, just as you need to know the nf9 elements that you use, you would have to know the semantics from the appropriate vendor. The collisions cannot be avoided, however, they are not managed in nf9 either.<br> <br> </div> As you say, there is currently no PEN for general experimental use that could be used as a fallback.</div> </div> </blockquote> <br> Even if there was, you shouldn't use such a PEN in a released product because it could conflict with someone else's usage.<br> <br> P.<br> <br> <br> <blockquote cite=3D"mid:[email protected]= ail.com" type=3D"cite"> <div dir=3D"ltr"> <div> I think that creating a specific PEN for this task is the right solution.<br> <br> </div> <div>Best regards,<br> </div> Petr<br> <div> <div> <div> <div> <div> <div> <div> <div class=3D"gmail_extra"><br> <div class=3D"gmail_quote">On Thu, Apr 2, 2015 at 10:36 PM, Gerhard Muenz <span dir=3D"ltr"><<= a moz-do-not-send=3D"true" href=3D"mailto:[email protected]" target=3D"_blank">[email protected]</a>&g= t;</span> wrote:<br> <blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <div bgcolor=3D"#FFFFFF" text=3D"#000000"> <b= r> Hi Petr,<br> <br> I do not understand the use case of the new PEN that you suggest.<br> <br> It does not make sense to register a PEN for an ID space that is not centrally managed in a kind of registry because uniqueness of ID usage must be ensured. Usually, the owner of a PEN maintains such a registry.<br> <br> Is there a registry that lists all vendor specific Netflow 9 Field Types? <br> If not, how can you be sure that there are no collisions in the ID space?<br> <br> I think that you need to find and use the PENs of the vendors that have defined the additional Netflow 9 Field Types IEs, regardless of whether the IDs are below or above 2^15. Unfortunately, there is no PEN for general experimental use (at least I have not found any) that you could use as a fallback if you do not find an appropriate vendor PEN.<br> <br> If you want to use your own non-standard IEs, then you should use the PEN of your organization:<br> <br> 8057<br> =C2=A0 CESNET<br> =C2=A0=C2=A0=C2=A0 CESNET masters team<br> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 masters&= <a moz-do-not-send=3D"true" href=3D"http://cesnet.cz" target=3D"_blan= k">cesnet.cz</a><br> <br> Maybe, you can also use this PEN to map Field Types for which you do not find a vendor.<br> <br> Regards,<br> Gerhard <div> <div class=3D"h5"><br> =C2=A0<br> <br> <div>On 26.03.2015 08:14, Petr Velan wrote:<br> </div> <blockquote type=3D"cite"> <div dir=3D"ltr"> <div> <div>Hi Andrew, all,<br> <br> </div> thank you for your explanation regarding nprobe. <br> <br> However, we also need a fallback for unknown exporters with IEs > 2^15. The generic requests for PENs need organization name, contact name and email address. I can try to request the PEN for NetFlow v9 compatibility myself, but I'd like it to be more public. Therefore, I suggest to complete the request with something like:<br> <b>Organization Name</b>: NetFlow v9 to IPFIX<br> <b>Contact Name</b>: IPFIX WG<br> <b>Contact E-Mail: </b><a moz-do-not-send=3D"true" href=3D"mailto:[email protected]" target=3D"_blank">[email protected]= g</a><br> <br> </div> <div>This is just a first proposal to get things moving, please add your thoughts. Once the PEN is granted, we can move forward and explain its purpose in a short RFC. <br> <br> </div> <div>Petr<br> </div> </div> <div class=3D"gmail_extra"><br> <div class=3D"gmail_quote">On Tue, Jan 6, 2015 at 9:07 PM, Andrew Feren <span dir=3D"ltr"><<a moz-do-not-send=3D"true" href=3D"mailto:andrewf@plixer= .com" target=3D"_blank">andrewf@pli= xer.com</a>></span> wrote:<br> <blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1= ex"> <div bgcolor=3D"#FFFFFF" text=3D"#000000"> <div>Hi Petr,<span><br> <br> On 01/06/2015 07:03 AM, Petr Velan wrote:<br> </span></div> <span> <blockquote type=3D"cite"> <div dir=3D"ltr"> <div> <div> <div> <div> <div> <div> <div>Hello all,<br> <br> </div> I'm not sure whether this is the right place to ask, but we encountered following problem when converting NetFlow v9 messages to IPFIX.<br> <br> </div> Some vendors (I've heard of ntop) are using elements IDs large than 32767 in NetFlow v9. When converting messages with these elements to IPFIX, they are considered to be Enterprise Numbers. To generate proper IPFIX message, we need to do one of the following:<br> </div> a) Generate a list of the elements and map them to PEN of the correct vendor. However, this would result in an attempt to cover all possible elements that anybody used in NetFlow v9. Moreover, we would still have to somehow handle the cases where the element is unknown<br> </div> </div> </div> </div> </div> </blockquote> </span> This should help with ntop/nprobe<br> <br> Recent versions of nprobe (since version 5.5.5 I think) all use the following mapping.<br> <br> PEN =3D 35632 and IPFIXID =3D (v9ID - 57472)<br> <br> For example, one v9 IE that nprobe exports is MYSQL_SERVER_VERSION 57667.=C2= =A0 The IPFIX equivalent would be<br> MYSQL_SERVER_VERSION(35632/195).<br> <br> The nprobe docs have a complete list.<br> <br> Older versions of nprobe (pre ~2010) use IEs not in RFC 3954, but later allocated in IANA.=C2=A0 Ther= e is no good way to convert those v9 exports to IPFIX.<span><fo= nt color=3D"#888888"><br> <br> -Andrew<br> <br> <br> </font></span> <blockquote type=3D"cite"><sp= an> <div dir=3D"ltr"> <div> <div> <div>b) Request a PEN for NetFlow compatibility and just add this PEN for every element that has ID larger than 32767.<br> <br> </div> Personally, I believe that the b) is more general and error-prone. Do you think, that it would be possible to dedicate whole PEN to this cause?<br> <br> </div> Thank you for any opinions,<br> </div> <br> Petr Velan<br> <div> <div><br> </div> </div> </div> <br> <fieldset></fieldset> <br> </span><span> <pre>____________________= ___________________________ IPFIX mailing list <a moz-do-not-send=3D"true" href=3D"mailto:[email protected]" target=3D"_bla= nk">[email protected]</a> <a moz-do-not-send=3D"true" href=3D"https://www.ietf.org/mailman/listinfo= /ipfix" target=3D"_blank">https://www.ietf.org/mailman/listinfo/ipfix</a> </pre> </span></blockquote> <br> </div> </blockquote> </div> <br> </div> <br> <fieldset></fieldset> <br> <pre>________________________________= _______________ IPFIX mailing list <a moz-do-not-send=3D"true" href=3D"mailto:[email protected]" target=3D"_bla= nk">[email protected]</a> <a moz-do-not-send=3D"true" href=3D"https://www.ietf.org/mailman/listinfo= /ipfix" target=3D"_blank">https://www.ietf.org/mailman/listinfo/ipfix</a> </pre> </blockquote> <br> </div> </div> </div> </blockquote> </div> <br> </div> </div> </div> </div> </div> </div> </div> </div> </div> </blockquote> <br> </body> </html> --------------040401040503000409030408-- --===============3640271284766209822== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ IPFIX mailing list [email protected] https://www.ietf.org/mailman/listinfo/ipfix --===============3640271284766209822==--