Re: Argus & IPFIX?

I'm using the latest stable version (argus-clients-3.0.8) downloaded from
the website (http://www.qosient.com/argus/downloads.shtml).

That command doesn't appear to be working upon running it, it appears to
just immediately exit and throw me back into a command prompt.

In comparison if I run the following:

ra -S cisco://192.xx.xx.xx:9995

It appears to stay running and sits on a new line with a flashing cursor
waiting to print something possibly?  But nothing is printed to stdout.

Thank you,

-Drew

On Tue, Oct 24, 2017 at 1:51 PM, Carter Bullard <[email protected]> wrote:

> Not sure what version you are using, but you should be able to read IPFIX
> data from an ra* program using the same strategy as radium.
> Does this work ???
>
>    ra -S ipfix://192.xx.xx.xx:9995
>
> Carter
>
>
> On Oct 24, 2017, at 1:25 PM, Drew Dixon <[email protected]> wrote:
>
> For us near real-time processing isn't really an issue- so at least for
> now I'll be aiming to leverage the second approach you've described which
> also sounds to be the most common with the scenario we are in at the
> moment.  I've been reading/studying the radium/rastream/racluster
> documentation/man pages etc. but still have questions related to
> configuration where there are gaps.
>
> Being that this approach will be collecting IPFIX data directed to my
> server which will be running radium|rastream|racluster etc.  I'm hoping
> someone can help me better understand/confirm the best configuration
> options combination available via radium in order to do so?  I may be
> overthinking/over-complicating this a bit but my main confusion is
> surrounding how to tell radium to listen for that IPFIX data properly.  I'm
> getting incoming data on udp/9995 which is confirmed via tcpdump...
>
> I'm currently telling radium to listen for what the docs refer to as Cisco
> Netflow (cmdline -C) but I am using radium.conf and the current settings
> I'm using in the config file are as follows:
>
> RADIUM_DAEMON=yes
> RADIUM_DEBUG_LEVEL=0
> RADIUM_MAR_STATUS_INTERVAL=60
> RADIUM_CISCONETFLOW_PORT=192.xx.xx.xx:9995
> RADIUM_ACCESS_PORT=561
> RADIUM_BIND_IP=127.0.0.1
> RADIUM_OUTPUT_FILE=/flowdata/radium/radium.out
> RADIUM_SETUSER_ID=argus
> RADIUM_SETGROUP_ID=argus
>
> Invoked with.... radium -f /etc/radium.conf
>
> Does radium log somewhere other than /var/log/messages?  Is it required to
> manually specify the ability to generate debug logs when compiling for that
> to be available? I'm not seeing anything mentioning that specifically via
> ./configure --help for argus-clients.
>
> I'm trying to test that radium is processing the data I've confirmed is
> coming via udp/9995 using rastream like:
>
> rastream -S 127.0.0.01:561 <http://127.0.0.1:561/> -w -
>
> When I invoke rastream in this fashion (as root from a root shell) should
> I not see the converted argus formatted data being spewed out to stdout in
> unidirectional argus format?
>
> I do see the log entries from radium confirming the connection from
> rastream, does this look normal?:
>
> Oct 24 12:56:16  radium: radium[14196]: 12:56:16.271560 connect from
> localhost[127.0.0.1]
> Oct 24 12:56:16  radium[14196]: 12:56:16.271560 connect from
> localhost[127.0.0.1]
> Oct 24 12:56:30  radium: radium[14196]: 12:56:30.781854
> ArgusCheckClientMessage: client localhost[127.0.0.1] sent DONE
> Oct 24 12:56:30  radium[14196]: 12:56:30.781854 ArgusCheckClientMessage:
> client localhost[127.0.0.1] sent DONE
>
> However, I'm not seeing anything being printed out to stdout though, also
> the output file I've configured via radium doesn't seem to be doing much
> (as far as I can tell at least), it's incrementing in size but not nearly
> to the degree I would expect but then again it does appear to be a DBase 3
> data file which appears to be a database that is showing a lot of records
> already according to the output of the file command:
>
> $ file /flowdata/radium/radium.out
> /flowdata/radium/radium.out: DBase 3 data file with memo(s) (2097152
> records)
>
> Should I be using another instance of radium to test the incoming data is
> being processed rather than trying to use rastream for that?  Does this
> make sense and/or is that doable?
>
> Still very much learning all these tools- help is greatly appreciated,
> many thanks in advance.
>
> -Drew
>
> ---------- Forwarded message ----------
> From: Carter Bullard <[email protected]>
> Date: Tue, Oct 17, 2017 at 10:10 AM
> Subject: Re: [ARGUS] Argus & IPFIX?
> To: Drew Dixon <[email protected]>
>
>
> Hey Drew,
>
> The open source clients have all the basic functionality to build a system
> that can handle complex asymmetric monitoring problems, but they not
> designed specifically to do this, nor are they designed to go very fast...
>
> If you have any open source argus questions, please send them to the
> public mailing list.
>
> If you want to merge asymmetric flows from multiple sources, the best
> approach is going to be dependent on how quickly you want to know about
> issues on the wire.  If time is important, you will want to use radium() to
> collect real-time streaming data, and then use programs like rabins() to
> aggregate the uni-directional streams into bi-directional streams, and keep
> the processing pipeline going.  There are time issues to work out, but the
> concepts in radium and rabins are what are needed to provide near-realtime
> multi-source aggregation.
>
> If near real-time processing is not an issue, then having radium collect
> the IPFIX data, covert it to argus data and use a program like rastream()
> to break the stream up into files, and then post process the files with
> racluster() to merge the records to form the bi-directional flow data.
> That is the most common way of doing what you’ve described.  The open
> source racluster() is a bit of a memory hog, so you’ll want give your
> system a bit of RAM, and then tune the process to handle the number of
> flows that you are processing.  So far in our experience, 100G doesn’t
> significantly increase the numbers of simultaneous flows, but rather
> supports bigger elephants co-existing with the traditional mix of data
> (research university networks).  You may find that in your specific
> network, that the flow data processing requirements aren’t much different
> than 10G, but …, then again you may be more like a wireless cellular
> network, where 100G really means 10x the flows seen at 10G … just all
> depends on your network design and customer base.
>
> Good luck, and keep us posted ….
> Carter
>
> [image: QoSient]        <http://qosient.com/>
> Carter Bullard  <[email protected]>• CTO
> 150 E 57th Street, Suite 12D
> <https://maps.google.com/?q=150+E+57th+Street,+Suite+12D+%0D+%0D+%0D+%0D+New+York,+New+York+10022&entry=gmail&source=g>
> New York, New York 10022
> <https://maps.google.com/?q=150+E+57th+Street,+Suite+12D+%0D+%0D+%0D+%0D+New+York,+New+York+10022&entry=gmail&source=g>
> -2795
> Phone +1.212.588.9133 • Mobile +1.917.497.9494
>
>
> On Mon, Oct 16, 2017 at 1:52 PM, Carter Bullard <[email protected]>
> wrote:
>
>> Hey Drew,
>> Argus should be able to read most/any IPFIX TCP/UDP data source, at least
>> that is the goal.  To that end, if you have some IPFIX data that the ra*
>> programs can’t read, I’ll spend some time making it work.  So if your using
>> Juniper, have it export UDP IPFIX, and we should be able to read them, as
>> the router advertises the templates in a reasonable timeframe, as we need
>> to see the templates before we can decode the records (really terrible
>> design flaw).
>>
>> We, of course recommend that you generate your own flow records rather
>> than read from integrated IPFIX, especially if you’re network is going
>> particularly fast.  QoSient has 1g, 10g, 40g and 100g argus sensor
>> appliances for sale, so if you’re looking to do the do for real, think
>> about generating your own data.
>>
>> Hope all is most excellent,
>> Carter
>>
>> [image: QoSient] <http://qosient.com/>
>> Carter Bullard <[email protected]> • CTO
>> 150 E 57th Street, Suite 12D
>> <https://maps.google.com/?q=150+E+57th+Street,+Suite+12D+%0D+%0D+%0D+%0D+New+York,+New+York+10022&entry=gmail&source=g>
>> New York, New York 10022
>> <https://maps.google.com/?q=150+E+57th+Street,+Suite+12D+%0D+%0D+%0D+%0D+New+York,+New+York+10022&entry=gmail&source=g>
>> -2795
>> Phone +1.212.588.9133 • Mobile +1.917.497.9494
>>
>>
>>
>> On Oct 16, 2017, at 11:18 AM, Drew Dixon <[email protected]> wrote:
>>
>> Hello,
>>
>> I'm wondering what the current status of Argus' support of reading IPFIX
>> and if there might be any relevant information/updates on that front which
>> someone could share?
>>
>> I did some quick searching online and see mention of IPFIX in relation to
>> Argus but nothing really stating that it's officially supported at this
>> time etc.
>>
>> Thank you!
>>
>> -Drew
>>
>>
>>
>
>