Re: Argus flows to Kafka

--000000000000c8f065059248cbbb
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

Hi Carter,

Its less about what new is wanted, and more just to optimize how to get
what we already have there.

Kafka is simply a message queue system.  In principle, Argus or radium
could directly send the logs to a Kafka 'topic' (maybe simply using the
defined collector id or something) that would be buffered for as long as
someone wanted their buffers to be.  Then other consumers can come along
and process the data.  In my use case, it allows me to run local Kafka
queues on my sensors which produce events as fast as it can, while letting
my consumers (who are far less tolerant to spikes in events) come along and
pick up those logs as they can.  Additionally, I'd like to avoid writing a
XML log to disk, only to have to tail that log in (and all that comes with
that), to convert it into JSON for Elastic.  If a tool can write directly
to a Kafka buffer, in any sort of structured format, Kafka can do all the
heavy lifting of managing the records from there and you can leave
downstream processing to something else.

And I'm personally intending to use ELK, but there are tons of applications
and processors out there that will happily process Kafka queues.  I know
you have been hesitant in the past to support lots and lots of output
formats and types and I would be too, since it would take away from the
core flow development, but it would be great for us who like Argus and want
to load data from directly into other tools.

Thanks,
Phil





On Tue, Sep 10, 2019 at 11:20 AM Carter Bullard <[email protected]> wrote:

> Hey Phil,
>
> There are a few groups that have done specific ML strategies using
> streaming argus data, Oak Ridge National Labs has an operational system,
> Situ, and a number of commercial entities have used argus as a part of
> their ML offerings, but I=E2=80=99m not sure if they are using Kafka =E2=
=80=A6 TensorFlow
> has always been the most common buzz word with these groups =E2=80=A6 sin=
ce
> TensorFlow and Kafka are a common pair of terms, I think some of these
> companies are probably doing Kafka streaming and Argus data, but not sure
> that anyone will tell you ...
>
> Is there something that Kafka would need from radium, or the argus-client=
s
> that isn=E2=80=99t already there ....  Is there a specific thing that Kaf=
ka wants
> in its streaming pipeline ??
>
> Carter
>
> > On Sep 10, 2019, at 8:08 AM, Phillip Deneault <[email protected]> wrote:
> >
> > Is there an ra* tool, or is anyone aware of a 3rd party tool, that can
> process argus output directly into Kafka?  Final stop would be an ELK
> database, but using Kafka would be a better middle ground from a
> performance and maintenance point of view.
> >
> > Thanks,
> > Phil
> >
> >
> > _______________________________________________
> > argus mailing list
> > [email protected]
> > https://pairlist1.pair.net/mailman/listinfo/argus
>
>

--000000000000c8f065059248cbbb
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div dir=3D"ltr"><div>Hi Carter,</div><div><br></div><div>=
Its less about what new is wanted, and more just to optimize how to get wha=
t we already have there.=C2=A0 </div><div><br></div><div></div><div>Kafka i=
s simply a message queue system.=C2=A0 In principle, Argus or radium could =
directly send the logs to a Kafka &#39;topic&#39; (maybe simply using the d=
efined collector id or something) that would be buffered for as long as som=
eone wanted their buffers to be.=C2=A0 Then other consumers can come along =
and process the data.=C2=A0 In my use case, it allows me to run local Kafka=
 queues on my sensors which produce events as fast as it can, while letting=
 my consumers (who are far less tolerant to spikes in events) come along an=
d pick up those logs as they can.=C2=A0 Additionally, I&#39;d like to avoid=
 writing a XML log to disk, only to have to tail that log in (and all that =
comes with that), to convert it into JSON for Elastic.=C2=A0 If a tool can =
write directly to a Kafka buffer, in any sort of structured format, Kafka c=
an do all the heavy lifting of managing the records from there and you can =
leave downstream processing to something else.<br></div><div><br></div><div=
>And I&#39;m personally intending to use ELK, but there are tons of applica=
tions and processors out there that will happily process Kafka queues.=C2=
=A0 I know you have been hesitant in the past to support lots and lots of o=
utput formats and types and I would be too, since it would take away from t=
he core flow development, but it would be great for us who like Argus and w=
ant to load data from directly into other tools.</div><div><br></div><div>T=
hanks,</div><div>Phil<br></div><div><br></div><div><br></div><div><br></div=
><div><br></div></div><br><div class=3D"gmail_quote"><div dir=3D"ltr" class=
=3D"gmail_attr">On Tue, Sep 10, 2019 at 11:20 AM Carter Bullard &lt;<a href=
=3D"mailto:[email protected]">[email protected]</a>&gt; wrote:<br></div><=
blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-l=
eft:1px solid rgb(204,204,204);padding-left:1ex">Hey Phil,<br>
<br>
There are a few groups that have done specific ML strategies using streamin=
g argus data, Oak Ridge National Labs has an operational system, Situ, and =
a number of commercial entities have used argus as a part of their ML offer=
ings, but I=E2=80=99m not sure if they are using Kafka =E2=80=A6 TensorFlow=
 has always been the most common buzz word with these groups =E2=80=A6 sinc=
e TensorFlow and Kafka are a common pair of terms, I think some of these co=
mpanies are probably doing Kafka streaming and Argus data, but not sure tha=
t anyone will tell you ...<br>
<br>
Is there something that Kafka would need from radium, or the argus-clients =
that isn=E2=80=99t already there ....=C2=A0 Is there a specific thing that =
Kafka wants in its streaming pipeline ??<br>
<br>
Carter<br>
<br>
&gt; On Sep 10, 2019, at 8:08 AM, Phillip Deneault &lt;<a href=3D"mailto:de=
[email protected]" target=3D"_blank">[email protected]</a>&gt; wrote:<br>
&gt; <br>
&gt; Is there an ra* tool, or is anyone aware of a 3rd party tool, that can=
 process argus output directly into Kafka?=C2=A0 Final stop would be an ELK=
 database, but using Kafka would be a better middle ground from a performan=
ce and maintenance point of view.<br>
&gt; <br>
&gt; Thanks,<br>
&gt; Phil<br>
&gt; <br>
&gt; <br>
&gt; _______________________________________________<br>
&gt; argus mailing list<br>
&gt; <a href=3D"mailto:[email protected]" target=3D"_blank">[email protected]=
om</a><br>
&gt; <a href=3D"https://pairlist1.pair.net/mailman/listinfo/argus" rel=3D"n=
oreferrer" target=3D"_blank">https://pairlist1.pair.net/mailman/listinfo/ar=
gus</a><br>
<br>
</blockquote></div><br clear=3D"all"><br></div>

--000000000000c8f065059248cbbb--