Re: Argus flows to Kafka
Carter Bullard <[email protected]> Wed, 11 Sep 2019 11:53:21 -0400
| Newsgroups | gmane.network.argus |
|---|---|
| Message-ID | <[email protected]> |
--Apple-Mail=_519A7E5F-8D8F-4484-86B1-894163055491 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Hey Phil, Yep, not sure why I coupled Kafka with ML so quickly, but that is the = context that I=E2=80=99ve seen Kafka in for a while, so I just tagged = the buzz words together. I went to the Kafka site, and quickly realized = that its just a message fabric, at least the way I think about things = =E2=80=A6 There are a few ZeroMQ transports of argus data out there, = Gloriad did that for a while, and they were quite happy, but these = groups aren=E2=80=99t contributing the code back to the argus project = :o( =E2=80=A6 I won=E2=80=99t be surprised if there is a Kafka producer = and client for argus data somewhere in the universe ... I am in favor of doing an argus source and client over Kafka, as well as = ZeroMQ and others that the community thinks is reasonable to do. = We=E2=80=99re going to have a major change in the web site by October = (design is already done, just need to move it to Joomla on the new = openargus.org <http://openargus.org/> site). I=E2=80=99ve scoped out 4 = basic projects that we=E2=80=99re going to do in 2020, kind of a result = of the commercial Argus efforts and Argus+Streaming is #3. =20 If you=E2=80=99re interested, lets see if we can=E2=80=99t get something = developed in a reasonable bit of time. Have radium.1 be a Kafka stream = processor, since its our argus stream processor. So if you want to = write to Kafka, use radium =E2=80=A6 if you want to read Kafka, use = radium and then all the apps that want to get data get if from radium = ???? Carter > On Sep 11, 2019, at 11:37 AM, Phillip Deneault <[email protected]> = wrote: >=20 > Hi Carter, >=20 > Its less about what new is wanted, and more just to optimize how to = get what we already have there.=20 >=20 > Kafka is simply a message queue system. In principle, Argus or radium = could directly send the logs to a Kafka 'topic' (maybe simply using the = defined collector id or something) that would be buffered for as long as = someone wanted their buffers to be. Then other consumers can come along = and process the data. In my use case, it allows me to run local Kafka = queues on my sensors which produce events as fast as it can, while = letting my consumers (who are far less tolerant to spikes in events) = come along and pick up those logs as they can. Additionally, I'd like = to avoid writing a XML log to disk, only to have to tail that log in = (and all that comes with that), to convert it into JSON for Elastic. If = a tool can write directly to a Kafka buffer, in any sort of structured = format, Kafka can do all the heavy lifting of managing the records from = there and you can leave downstream processing to something else. >=20 > And I'm personally intending to use ELK, but there are tons of = applications and processors out there that will happily process Kafka = queues. I know you have been hesitant in the past to support lots and = lots of output formats and types and I would be too, since it would take = away from the core flow development, but it would be great for us who = like Argus and want to load data from directly into other tools. >=20 > Thanks, > Phil >=20 >=20 >=20 >=20 >=20 > On Tue, Sep 10, 2019 at 11:20 AM Carter Bullard <[email protected] = <mailto:[email protected]>> wrote: > Hey Phil, >=20 > There are a few groups that have done specific ML strategies using = streaming argus data, Oak Ridge National Labs has an operational system, = Situ, and a number of commercial entities have used argus as a part of = their ML offerings, but I=E2=80=99m not sure if they are using Kafka =E2=80= =A6 TensorFlow has always been the most common buzz word with these = groups =E2=80=A6 since TensorFlow and Kafka are a common pair of terms, = I think some of these companies are probably doing Kafka streaming and = Argus data, but not sure that anyone will tell you ... >=20 > Is there something that Kafka would need from radium, or the = argus-clients that isn=E2=80=99t already there .... Is there a specific = thing that Kafka wants in its streaming pipeline ?? >=20 > Carter >=20 > > On Sep 10, 2019, at 8:08 AM, Phillip Deneault <[email protected] = <mailto:[email protected]>> wrote: > >=20 > > Is there an ra* tool, or is anyone aware of a 3rd party tool, that = can process argus output directly into Kafka? Final stop would be an = ELK database, but using Kafka would be a better middle ground from a = performance and maintenance point of view. > >=20 > > Thanks, > > Phil > >=20 > >=20 > > _______________________________________________ > > argus mailing list > > [email protected] <mailto:[email protected]> > > https://pairlist1.pair.net/mailman/listinfo/argus = <https://pairlist1.pair.net/mailman/listinfo/argus> >=20 >=20 >=20 --Apple-Mail=_519A7E5F-8D8F-4484-86B1-894163055491 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 <html><head><meta http-equiv=3D"Content-Type" content=3D"text/html; = charset=3Dutf-8"></head><body style=3D"word-wrap: break-word; = -webkit-nbsp-mode: space; line-break: after-white-space;" class=3D"">Hey = Phil,<div class=3D"">Yep, not sure why I coupled Kafka with ML so = quickly, but that is the context that I=E2=80=99ve seen Kafka in for a = while, so I just tagged the buzz words together. I went to the = Kafka site, and quickly realized that its just a message fabric, at = least the way I think about things =E2=80=A6 There are a few ZeroMQ = transports of argus data out there, Gloriad did that for a while, and = they were quite happy, but these groups aren=E2=80=99t contributing the = code back to the argus project :o( =E2=80=A6 I won=E2=80=99t be = surprised if there is a Kafka producer and client for argus data = somewhere in the universe ...</div><div class=3D""><br = class=3D""></div><div class=3D"">I am in favor of doing an argus source = and client over Kafka, as well as ZeroMQ and others that the community = thinks is reasonable to do. We=E2=80=99re going to have a major = change in the web site by October (design is already done, just need to = move it to Joomla on the new <a href=3D"http://openargus.org" = class=3D"">openargus.org</a> site). I=E2=80=99ve scoped out 4 = basic projects that we=E2=80=99re going to do in 2020, kind of a result = of the commercial Argus efforts and Argus+Streaming is #3. = </div><div class=3D""><br class=3D""></div><div class=3D"">If = you=E2=80=99re interested, lets see if we can=E2=80=99t get something = developed in a reasonable bit of time. Have radium.1 be a Kafka = stream processor, since its our argus stream processor. So if you = want to write to Kafka, use radium =E2=80=A6 if you want to read Kafka, = use radium and then all the apps that want to get data get if from = radium ????</div><div class=3D""><br class=3D""><div = class=3D"">Carter</div><div class=3D""><br class=3D""><div><br = class=3D""><blockquote type=3D"cite" class=3D""><div class=3D"">On Sep = 11, 2019, at 11:37 AM, Phillip Deneault <<a = href=3D"mailto:[email protected]" class=3D"">[email protected]</a>> = wrote:</div><br class=3D"Apple-interchange-newline"><div class=3D""><div = dir=3D"ltr" class=3D""><div dir=3D"ltr" class=3D""><div class=3D"">Hi = Carter,</div><div class=3D""><br class=3D""></div><div class=3D"">Its = less about what new is wanted, and more just to optimize how to get what = we already have there. </div><div class=3D""><br = class=3D""></div><div class=3D""></div><div class=3D"">Kafka is simply a = message queue system. In principle, Argus or radium could directly = send the logs to a Kafka 'topic' (maybe simply using the defined = collector id or something) that would be buffered for as long as someone = wanted their buffers to be. Then other consumers can come along = and process the data. In my use case, it allows me to run local = Kafka queues on my sensors which produce events as fast as it can, while = letting my consumers (who are far less tolerant to spikes in events) = come along and pick up those logs as they can. Additionally, I'd = like to avoid writing a XML log to disk, only to have to tail that log = in (and all that comes with that), to convert it into JSON for = Elastic. If a tool can write directly to a Kafka buffer, in any = sort of structured format, Kafka can do all the heavy lifting of = managing the records from there and you can leave downstream processing = to something else.<br class=3D""></div><div class=3D""><br = class=3D""></div><div class=3D"">And I'm personally intending to use = ELK, but there are tons of applications and processors out there that = will happily process Kafka queues. I know you have been hesitant = in the past to support lots and lots of output formats and types and I = would be too, since it would take away from the core flow development, = but it would be great for us who like Argus and want to load data from = directly into other tools.</div><div class=3D""><br class=3D""></div><div = class=3D"">Thanks,</div><div class=3D"">Phil<br class=3D""></div><div = class=3D""><br class=3D""></div><div class=3D""><br class=3D""></div><div = class=3D""><br class=3D""></div><div class=3D""><br = class=3D""></div></div><br class=3D""><div class=3D"gmail_quote"><div = dir=3D"ltr" class=3D"gmail_attr">On Tue, Sep 10, 2019 at 11:20 AM Carter = Bullard <<a href=3D"mailto:[email protected]" = class=3D"">[email protected]</a>> wrote:<br = class=3D""></div><blockquote class=3D"gmail_quote" style=3D"margin:0px = 0px 0px 0.8ex;border-left:1px solid = rgb(204,204,204);padding-left:1ex">Hey Phil,<br class=3D""> <br class=3D""> There are a few groups that have done specific ML strategies using = streaming argus data, Oak Ridge National Labs has an operational system, = Situ, and a number of commercial entities have used argus as a part of = their ML offerings, but I=E2=80=99m not sure if they are using Kafka =E2=80= =A6 TensorFlow has always been the most common buzz word with these = groups =E2=80=A6 since TensorFlow and Kafka are a common pair of terms, = I think some of these companies are probably doing Kafka streaming and = Argus data, but not sure that anyone will tell you ...<br class=3D""> <br class=3D""> Is there something that Kafka would need from radium, or the = argus-clients that isn=E2=80=99t already there .... Is there a = specific thing that Kafka wants in its streaming pipeline ??<br = class=3D""> <br class=3D""> Carter<br class=3D""> <br class=3D""> > On Sep 10, 2019, at 8:08 AM, Phillip Deneault <<a = href=3D"mailto:[email protected]" target=3D"_blank" = class=3D"">[email protected]</a>> wrote:<br class=3D""> > <br class=3D""> > Is there an ra* tool, or is anyone aware of a 3rd party tool, that = can process argus output directly into Kafka? Final stop would be = an ELK database, but using Kafka would be a better middle ground from a = performance and maintenance point of view.<br class=3D""> > <br class=3D""> > Thanks,<br class=3D""> > Phil<br class=3D""> > <br class=3D""> > <br class=3D""> > _______________________________________________<br class=3D""> > argus mailing list<br class=3D""> > <a href=3D"mailto:[email protected]" target=3D"_blank" = class=3D"">[email protected]</a><br class=3D""> > <a href=3D"https://pairlist1.pair.net/mailman/listinfo/argus" = rel=3D"noreferrer" target=3D"_blank" = class=3D"">https://pairlist1.pair.net/mailman/listinfo/argus</a><br = class=3D""> <br class=3D""> </blockquote></div><br clear=3D"all" class=3D""><br class=3D""></div> </div></blockquote></div><br class=3D""></div></div></body></html>= --Apple-Mail=_519A7E5F-8D8F-4484-86B1-894163055491--