Re: Argus flows to Kafka

--00000000000087d2e705924b57b3
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

On Wed, Sep 11, 2019 at 11:53 AM Carter Bullard <[email protected]> wrote:

> If you=E2=80=99re interested, lets see if we can=E2=80=99t get something =
developed in a
> reasonable bit of time.  Have radium.1 be a Kafka stream processor, since
> its our argus stream processor.  So if you want to write to Kafka, use
> radium =E2=80=A6 if you want to read Kafka, use radium and then all the a=
pps that
> want to get data get if from radium ???
>

I think it would be really great if radium produced flows to Kafka.  There
are others who would also be interested in using something like that.
I'm not sure how much utility there would be in the use case to consume
data from Kafka into ra* tools, but you are more tapped into the
flow-based-ecosystem and your own customers than I am.  Personally, I've
found the ra* tools efficient with the rate of record creation in Argus.
The only thing Kafka might buy you is robustness in case of disruption of
network traffic between control points.

Thanks,
Phil




> Carter
>
>
> On Sep 11, 2019, at 11:37 AM, Phillip Deneault <[email protected]> wrote:
>
> Hi Carter,
>
> Its less about what new is wanted, and more just to optimize how to get
> what we already have there.
>
> Kafka is simply a message queue system.  In principle, Argus or radium
> could directly send the logs to a Kafka 'topic' (maybe simply using the
> defined collector id or something) that would be buffered for as long as
> someone wanted their buffers to be.  Then other consumers can come along
> and process the data.  In my use case, it allows me to run local Kafka
> queues on my sensors which produce events as fast as it can, while lettin=
g
> my consumers (who are far less tolerant to spikes in events) come along a=
nd
> pick up those logs as they can.  Additionally, I'd like to avoid writing =
a
> XML log to disk, only to have to tail that log in (and all that comes wit=
h
> that), to convert it into JSON for Elastic.  If a tool can write directly
> to a Kafka buffer, in any sort of structured format, Kafka can do all the
> heavy lifting of managing the records from there and you can leave
> downstream processing to something else.
>
> And I'm personally intending to use ELK, but there are tons of
> applications and processors out there that will happily process Kafka
> queues.  I know you have been hesitant in the past to support lots and lo=
ts
> of output formats and types and I would be too, since it would take away
> from the core flow development, but it would be great for us who like Arg=
us
> and want to load data from directly into other tools.
>
> Thanks,
> Phil
>
>
>
>
>
> On Tue, Sep 10, 2019 at 11:20 AM Carter Bullard <[email protected]>
> wrote:
>
>> Hey Phil,
>>
>> There are a few groups that have done specific ML strategies using
>> streaming argus data, Oak Ridge National Labs has an operational system,
>> Situ, and a number of commercial entities have used argus as a part of
>> their ML offerings, but I=E2=80=99m not sure if they are using Kafka =E2=
=80=A6 TensorFlow
>> has always been the most common buzz word with these groups =E2=80=A6 si=
nce
>> TensorFlow and Kafka are a common pair of terms, I think some of these
>> companies are probably doing Kafka streaming and Argus data, but not sur=
e
>> that anyone will tell you ...
>>
>> Is there something that Kafka would need from radium, or the
>> argus-clients that isn=E2=80=99t already there ....  Is there a specific=
 thing that
>> Kafka wants in its streaming pipeline ??
>>
>> Carter
>>
>> > On Sep 10, 2019, at 8:08 AM, Phillip Deneault <[email protected]> wrote:
>> >
>> > Is there an ra* tool, or is anyone aware of a 3rd party tool, that can
>> process argus output directly into Kafka?  Final stop would be an ELK
>> database, but using Kafka would be a better middle ground from a
>> performance and maintenance point of view.
>> >
>> > Thanks,
>> > Phil
>> >
>> >
>> > _______________________________________________
>> > argus mailing list
>> > [email protected]
>> > https://pairlist1.pair.net/mailman/listinfo/argus
>>
>>
>
>
>

--00000000000087d2e705924b57b3
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div class=3D"gmail_quote"><div dir=3D"ltr" class=3D"gmail=
_attr">On Wed, Sep 11, 2019 at 11:53 AM Carter Bullard &lt;<a href=3D"mailt=
o:[email protected]">[email protected]</a>&gt; wrote:<br></div><blockquot=
e class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px s=
olid rgb(204,204,204);padding-left:1ex"><div style=3D"overflow-wrap: break-=
word;"><div>If you=E2=80=99re interested, lets see if we can=E2=80=99t get =
something developed in a reasonable bit of time.=C2=A0 Have radium.1 be a K=
afka stream processor, since its our argus stream processor.=C2=A0 So if yo=
u want to write to Kafka, use radium =E2=80=A6 if you want to read Kafka, u=
se radium and then all the apps that want to get data get if from radium ??=
?<br></div></div></blockquote><div><br></div><div>I think it would be reall=
y great if radium produced flows to Kafka.=C2=A0 There are others who would=
 also be interested in using something like that.<br></div><div>I&#39;m not=
 sure how much utility there would be in the use case to consume data from =
Kafka into ra* tools, but you are more tapped into the flow-based-ecosystem=
 and your own customers than I am.=C2=A0 Personally, I&#39;ve found the ra*=
 tools efficient with the rate of record creation in Argus.=C2=A0 The only =
thing Kafka might buy you is robustness in case of disruption of network tr=
affic between control points.<br></div><div><br></div><div>Thanks,</div><di=
v>Phil<br></div><div><br></div><div><br></div><div>=C2=A0</div><blockquote =
class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px sol=
id rgb(204,204,204);padding-left:1ex"><div style=3D"overflow-wrap: break-wo=
rd;"><div><div>Carter</div><div><br><div><br><blockquote type=3D"cite"><div=
>On Sep 11, 2019, at 11:37 AM, Phillip Deneault &lt;<a href=3D"mailto:denea=
[email protected]" target=3D"_blank">[email protected]</a>&gt; wrote:</div><br class=
=3D"gmail-m_-518548705328075443Apple-interchange-newline"><div><div dir=3D"=
ltr"><div dir=3D"ltr"><div>Hi Carter,</div><div><br></div><div>Its less abo=
ut what new is wanted, and more just to optimize how to get what we already=
 have there.=C2=A0 </div><div><br></div><div></div><div>Kafka is simply a m=
essage queue system.=C2=A0 In principle, Argus or radium could directly sen=
d the logs to a Kafka &#39;topic&#39; (maybe simply using the defined colle=
ctor id or something) that would be buffered for as long as someone wanted =
their buffers to be.=C2=A0 Then other consumers can come along and process =
the data.=C2=A0 In my use case, it allows me to run local Kafka queues on m=
y sensors which produce events as fast as it can, while letting my consumer=
s (who are far less tolerant to spikes in events) come along and pick up th=
ose logs as they can.=C2=A0 Additionally, I&#39;d like to avoid writing a X=
ML log to disk, only to have to tail that log in (and all that comes with t=
hat), to convert it into JSON for Elastic.=C2=A0 If a tool can write direct=
ly to a Kafka buffer, in any sort of structured format, Kafka can do all th=
e heavy lifting of managing the records from there and you can leave downst=
ream processing to something else.<br></div><div><br></div><div>And I&#39;m=
 personally intending to use ELK, but there are tons of applications and pr=
ocessors out there that will happily process Kafka queues.=C2=A0 I know you=
 have been hesitant in the past to support lots and lots of output formats =
and types and I would be too, since it would take away from the core flow d=
evelopment, but it would be great for us who like Argus and want to load da=
ta from directly into other tools.</div><div><br></div><div>Thanks,</div><d=
iv>Phil<br></div><div><br></div><div><br></div><div><br></div><div><br></di=
v></div><br><div class=3D"gmail_quote"><div dir=3D"ltr" class=3D"gmail_attr=
">On Tue, Sep 10, 2019 at 11:20 AM Carter Bullard &lt;<a href=3D"mailto:car=
[email protected]" target=3D"_blank">[email protected]</a>&gt; wrote:<br></d=
iv><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;bord=
er-left:1px solid rgb(204,204,204);padding-left:1ex">Hey Phil,<br>
<br>
There are a few groups that have done specific ML strategies using streamin=
g argus data, Oak Ridge National Labs has an operational system, Situ, and =
a number of commercial entities have used argus as a part of their ML offer=
ings, but I=E2=80=99m not sure if they are using Kafka =E2=80=A6 TensorFlow=
 has always been the most common buzz word with these groups =E2=80=A6 sinc=
e TensorFlow and Kafka are a common pair of terms, I think some of these co=
mpanies are probably doing Kafka streaming and Argus data, but not sure tha=
t anyone will tell you ...<br>
<br>
Is there something that Kafka would need from radium, or the argus-clients =
that isn=E2=80=99t already there ....=C2=A0 Is there a specific thing that =
Kafka wants in its streaming pipeline ??<br>
<br>
Carter<br>
<br>
&gt; On Sep 10, 2019, at 8:08 AM, Phillip Deneault &lt;<a href=3D"mailto:de=
[email protected]" target=3D"_blank">[email protected]</a>&gt; wrote:<br>
&gt; <br>
&gt; Is there an ra* tool, or is anyone aware of a 3rd party tool, that can=
 process argus output directly into Kafka?=C2=A0 Final stop would be an ELK=
 database, but using Kafka would be a better middle ground from a performan=
ce and maintenance point of view.<br>
&gt; <br>
&gt; Thanks,<br>
&gt; Phil<br>
&gt; <br>
&gt; <br>
&gt; _______________________________________________<br>
&gt; argus mailing list<br>
&gt; <a href=3D"mailto:[email protected]" target=3D"_blank">[email protected]=
om</a><br>
&gt; <a href=3D"https://pairlist1.pair.net/mailman/listinfo/argus" rel=3D"n=
oreferrer" target=3D"_blank">https://pairlist1.pair.net/mailman/listinfo/ar=
gus</a><br>
<br>
</blockquote></div><br clear=3D"all"><br></div>
</div></blockquote></div><br></div></div></div></blockquote></div><br clear=
=3D"all"><br></div>

--00000000000087d2e705924b57b3--