Re: Argus & IPFIX?
Drew Dixon <[email protected]> Mon, 21 May 2018 15:04:20 -0400
| Newsgroups | gmane.network.argus |
|---|---|
| Message-ID | <CA+pNqTOBRttGMd=79CpKGQuvGoxRq1EvN5tUxhUHc0MdYEx8pw@mail.gmail.com> |
Hey Carter,
Thanks for quick response- so I think I see what you're getting at here,
but just confirming that you're saying the OutBytes/OutPkts you commented
out in the original V9 code because most v9/IPFIX data sources (e.g.
routers) don't generate metrics for both sides of the conversation
typically, which makes sense. If so, I probably made some incorrect
assumptions- please allow me back up here a bit-
So the problem I have been trying to solve is now that we are collecting
100% IPFIX I am not seeing any metrics for Dst Bytes or Dst Packets when
reading our IPFIX converted argus formatted data. So for example, when we
were in limbo of getting collection of 100% IPFIX going we were temporarily
collecting some Netflow v5 data, when collecting this data and converting
to argus format we definitely had both source bytes/packets AND destination
bytes/packets.
Here's a quick example of when we only had Netflow v5 data:
racount -r ./argus.2018.01.15.12.00.01.xz
racount records total_pkts src_pkts dst_pkts
total_bytes src_bytes dst_bytes
sum 10769234 1185001190 813545312 371455878
1066725556584 709307666644 357417889940
Here is the same type of example when we now only collect IPFIX data:
racount -r ./argus.2018.05.21.12.00.01.xz
racount records total_pkts src_pkts *dst_pkts *
total_bytes src_bytes *dst_bytes*
sum 9817964 826668156 826668156 *0 *
734433831158 734433831158 *0*
racount -r ./argus.2018.05.21.12.00.01.xz ./argus.2018.05.21.12.05.01.xz
./argus.2018.05.21.12.10.01.xz
racount records total_pkts src_pkts *dst_pkts*
total_bytes src_bytes *dst_bytes*
sum 29443962 2519379651 2519379651 *0*
2280581037429 2280581037429 *0*
FYI, these .xz files above have already been processed through racluster
using the default options prior to being archived. (radium -> radium.out
file -> racluster -> xz) Using 5 minute archives.
Doesn't seem to matter how many racluster processed archives I read I still
don't see any dst pkts or dst bytes...so trying to figure out why that
is...also same thing if I read the radium.out file before it gets processed
by racluster (which might make more sense?).
racount -r radium.out
racount records total_pkts src_pkts *dst_pkts*
total_bytes src_bytes *dst_bytes*
sum 10042725 467428485 467428485 *0*
431751989943 431751989943 *0*
-Drew
On Mon, May 21, 2018 at 1:58 PM, Carter Bullard <[email protected]> wrote:
> Hey Drew,
> Are you sure that your IPFIX source generates bi-directional metrics ???
> I would be surprised … and why the OutBytes is commented out. But it seems
> that if you uncomment this code, no errors should arise.
> There is no sysUptime in V10, and there isn’t an assignment for that
> variable, maybe you uncommitted incorrectly ???
>
> Carter
>
>
> On May 21, 2018, at 1:21 PM, Drew Dixon <[email protected]> wrote:
>
> Hi Carter,
>
> Reviving this older thread- seems I may have been getting ahead of myself
> with some of the things I've been testing around with Argus client tools.
> So if you recall we worked on modifying the argus client tools code to
> support IPFIX (mostly, up to Netflow v9 field/element level compatibility
> that was already built into the code). Upon fully switching over to
> collecting 100% IPFIX flow data from our routers I realized that there were
> still some artifacts of making that work that I did not notice after we got
> the issue resolved with IPFIX timestamps.
>
> I noticed in the original untouched copy of the client tools V9 code in
> argus_import.c that you had the k_CiscoV9OutBytes and k_CiscoV9OutPkts code
> commented out:
>
> *****************************************************************
> case k_CiscoV9OutBytes: {
> /*
> canon->metric.dst.bytes = value.val32[0];
> ns->dsrindex |= 1 << ARGUS_METRIC_INDEX;
> ns->dsrs[ARGUS_METRIC_INDEX] =
> &canon->metric.hdr;
> */
> break;
> }
> case k_CiscoV9OutPkts: {
> /*
> canon->metric.dst.pkts = value.val32[0];
> ns->dsrindex |= 1 << ARGUS_METRIC_INDEX;
> ns->dsrs[ARGUS_METRIC_INDEX] =
> &canon->metric.hdr;
> */
> break;
> *****************************************************************
>
> So this being commented out also made it's way into our V10 version if
> this code, however we really need the dst bytes and dst packets to be
> parsed by radium when reading our IPFIX data coming in off the wire. For
> some reason, we apparently also had to comment out everything in
> k_CiscoV10LastSwitched and k_CiscoV10FirstSwitched, probably to get it to
> compile without errors. I attempted to uncomment out k_CiscoV10OutBytes
> and k_CiscoV10OutPkts and also the k_CiscoV10LastSwitched and
> k_CiscoV10FirstSwitched code we had commented out (but FWIW k_CiscoV*9*LastSwitched
> and k_CiscoV*9*FirstSwitched is not commented out in your original V9
> code).
>
> I'm getting the following errors, after attempting to uncomment this out
> in our V10 version of the code and was wondering if someone could help me
> to get this going so radium parses out the dst bytes and dst packets or if
> this is a larger problem to tackle since the original V9 code had been
> already commented out?
>
> *****************************************************************
> ./argus_import.c: In function ‘ArgusParseCiscoRecordV10Data’:
> ./argus_import.c:2082:69: error: ‘CiscoFlowHeaderV10_t’ has no member
> named ‘sysUptime’
> secs = ((timeval -
> ArgusNetFlow->sysUptime) / 1000);
> ^
> ./argus_import.c:2083:69: error: ‘CiscoFlowHeaderV10_t’ has no member
> named ‘sysUptime’
> usecs = ((timeval -
> ArgusNetFlow->sysUptime) % 1000) * 1000;
> ^
> ./argus_import.c:2101:69: error: ‘CiscoFlowHeaderV10_t’ has no member
> named ‘sysUptime’
> secs = ((timeval -
> ArgusNetFlow->sysUptime) / 1000);
> ^
> ./argus_import.c:2102:69: error: ‘CiscoFlowHeaderV10_t’ has no member
> named ‘sysUptime’
> usecs = ((timeval -
> ArgusNetFlow->sysUptime) % 1000) * 1000;
> ^
> ./argus_import.c: In function ‘ArgusReadCiscoDatagramSocket’:
> ./argus_import.c:4164:34: error: ‘CiscoFlowHeaderV10_t’ has no member
> named ‘sysUptime’
> ArgusNetFlow->sysUptime =
> ntohl(ArgusNetFlow->sysUptime);
> ^
> In file included from ../include/argus_os.h:106:0,
> from ../include/argus_compat.h:94,
> from ./argus_import.c:49:
> ./argus_import.c:4164:73: error: ‘CiscoFlowHeaderV10_t’ has no member
> named ‘sysUptime’
> ArgusNetFlow->sysUptime =
> ntohl(ArgusNetFlow->sysUptime);
> ^
> ./argus_import.c:4165:34: error: ‘CiscoFlowHeaderV10_t’ has no member
> named ‘unix_secs’
> ArgusNetFlow->unix_secs =
> ntohl(ArgusNetFlow->unix_secs);
> ^
> In file included from ../include/argus_os.h:106:0,
> from ../include/argus_compat.h:94,
> from ./argus_import.c:49:
> ./argus_import.c:4165:73: error: ‘CiscoFlowHeaderV10_t’ has no member
> named ‘unix_secs’
> ArgusNetFlow->unix_secs =
> ntohl(ArgusNetFlow->unix_secs);
> ^
> make[1]: *** [argus_import.o] Error 1
> *****************************************************************
>
> Thank you greatly,
>
> -Drew
>
>
> On Thu, Oct 26, 2017 at 2:08 PM, Carter Bullard <[email protected]>
> wrote:
>
>> Hey Drew,
>> Well, I’ve been working on some non-critical Argus bugs for well over 2
>> years now, as I have absolutely little time for free engineering anymore.
>> Getting IPFIX data into open source Argus is unfortunately at the same
>> priority level as those bugs, so you may have to wait for a while for me to
>> get to it. I was willing to fix a feature thinking it was already in
>> Argus, rather than doing new work. But it isn’t hard to do, maybe someone
>> else has done the work and can share ???? I heard a rumor that some at the
>> CMU’s CERT have been doing some Argus integration work for Silk, which is
>> mostly IPFIX. Maybe someone there has taken on this little project ???
>>
>> Carter
>>
>>
>> On Oct 26, 2017, at 12:31 PM, Drew Dixon <[email protected]> wrote:
>>
>> : ( Well, I appreciate your help anyhow, it happens...I might be able to
>> figure something out to convert the IPFIX to Netflow v9 temporarily I
>> guess...would be getting really messy and wasting CPU cycles/disk space
>> trying to do that tho...I thought you said "To that end, if you have some
>> IPFIX data that the ra* programs can’t read, I’ll spend some time making it
>> work." though? I suppose I was banking on that either way.
>>
>> Thanks,
>>
>> -Drew
>>
>> On Wed, Oct 25, 2017 at 7:15 PM, Carter Bullard <[email protected]>
>> wrote:
>>
>>> Hey Drew,
>>> So, all the records are IPFIX, my mistake, we only go to Netflow V9 in
>>> argus-clients-3.0.8.2 …. we see that the version number is 10, and just
>>> jump right past the payload. Any chance you can get your box to output
>>> Netflow V9 ???? My memory was on our commercial clients, which do a lot,
>>> netflow, sflow, ipfix.
>>>
>>> Sorry to have wasted all your time … If you get motivated, the code to
>>> support Netflow V10 would go in argus_import.c, create the NetflowV10
>>> routines using the NetflowV9 support as a guide (they are almost
>>> identical), add some includes and constants and it should be difficult,
>>> given that v10 is almost identical to v9. We’ll put it in the distribution
>>> if you get it going ...
>>>
>>> Really sorry about that !!!!
>>> Carter
>>>
>>>
>>>>>>>>
>>>>>>>> Carter Bullard <[email protected]>• CTO
>>>>>>>> 150 E 57th Street, Suite 12D
>>>>>>>> <https://maps.google.com/?q=150+E+57th+Street,+Suite+12D+%0D+%0D+%0D+%0D+New+York,+New+York+10022&entry=gmail&source=g>
>>>>>>>> New York, New York 10022
>>>>>>>> <https://maps.google.com/?q=150+E+57th+Street,+Suite+12D+%0D+%0D+%0D+%0D+New+York,+New+York+10022&entry=gmail&source=g>
>>>>>>>> -2795
>>>>>>>> Phone +1.212.588.9133 • Mobile +1.917.497.9494
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Oct 16, 2017 at 1:52 PM, Carter Bullard <[email protected]
>>>>>>>> > wrote:
>>>>>>>>
>>>>>>>>> Hey Drew,
>>>>>>>>> Argus should be able to read most/any IPFIX TCP/UDP data source,
>>>>>>>>> at least that is the goal. To that end, if you have some IPFIX data that
>>>>>>>>> the ra* programs can’t read, I’ll spend some time making it work. So if
>>>>>>>>> your using Juniper, have it export UDP IPFIX, and we should be able to read
>>>>>>>>> them, as the router advertises the templates in a reasonable timeframe, as
>>>>>>>>> we need to see the templates before we can decode the records (really
>>>>>>>>> terrible design flaw).
>>>>>>>>>
>>>>>>>>> We, of course recommend that you generate your own flow records
>>>>>>>>> rather than read from integrated IPFIX, especially if you’re network is
>>>>>>>>> going particularly fast. QoSient has 1g, 10g, 40g and 100g argus sensor
>>>>>>>>> appliances for sale, so if you’re looking to do the do for real, think
>>>>>>>>> about generating your own data.
>>>>>>>>>
>>>>>>>>> Hope all is most excellent,
>>>>>>>>> Carter
>>>>>>>>>
>>>>>>>>> [image: QoSient] <http://qosient.com/>
>>>>>>>>> Carter Bullard <[email protected]> • CTO
>>>>>>>>> 150 E 57th Street, Suite 12D
>>>>>>>>> <https://maps.google.com/?q=150+E+57th+Street,+Suite+12D+%0D+%0D+%0D+%0D+New+York,+New+York+10022&entry=gmail&source=g>
>>>>>>>>> New York, New York 10022
>>>>>>>>> <https://maps.google.com/?q=150+E+57th+Street,+Suite+12D+%0D+%0D+%0D+%0D+New+York,+New+York+10022&entry=gmail&source=g>
>>>>>>>>> -2795
>>>>>>>>> Phone +1.212.588.9133 • Mobile +1.917.497.9494
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Oct 16, 2017, at 11:18 AM, Drew Dixon <[email protected]>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> Hello,
>>>>>>>>>
>>>>>>>>> I'm wondering what the current status of Argus' support of reading
>>>>>>>>> IPFIX and if there might be any relevant information/updates on that front
>>>>>>>>> which someone could share?
>>>>>>>>>
>>>>>>>>> I did some quick searching online and see mention of IPFIX in
>>>>>>>>> relation to Argus but nothing really stating that it's officially supported
>>>>>>>>> at this time etc.
>>>>>>>>>
>>>>>>>>> Thank you!
>>>>>>>>>
>>>>>>>>> -Drew
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>
>