Re: Attribute clarification
[email protected] Fri, 6 Sep 2019 13:26:36 -0400
| Newsgroups | gmane.network.argus |
|---|---|
| Message-ID | <[email protected]> |
--Apple-Mail=_55699C12-DB11-4F7D-BFF1-C03743C568FB Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 I think that there maybe some information in knowing the distance = between argus records, but the variables that impact the record distance = is pretty independent of anything that I would think of interest. = Traffic mix, argus configuration, etc =E2=80=A6. And in high volume = traffic sources, the counter rolls over regularly so long term = behavioral trends would be confused by the number, I suspect, but that = can be dealt with, without much issue =E2=80=A6. I think that some want to use the sequence number as a quasi key for the = transaction, but we have other transaction ids that are designed for = that =E2=80=A6 and of course the flow key, plus time is a much better = discriminator for flow identification. Hope all is most excellent, Carter =20 > On Sep 6, 2019, at 10:47 AM, Giampaolo Bovenzi = <[email protected]> wrote: >=20 > Hi Carter, > =20 > thank for the rapidity and precision of your response. > I have got no problem with the attribute itself. > =20 > My curiosity comes from using this attribute to conduct network = traffic modeling. > As I have seen, some researchers are using the Argus seq number as = input to a modeling system, but, given your response, this attribute = introduces a bias and should be discarded to model network traffic. > =20 > Thanks again and good work! > GB > =20 > Da: [email protected] <mailto:[email protected]> > Inviato: venerd=C3=AC 6 settembre 2019 15:56 > A: Argus <mailto:[email protected]> > Oggetto: Re: [ARGUS] Attribute clarification > =20 > Hey Giampaolo, > The Argus sequence number is a monotonically increasing record number = in the ARGUS_TRANSPORT_DSR, the structure in the argus record that is = used to identify the data source of data during data transport between = nodes. The 32-bit int seqnum is generated by the originating Argus data = source, and is used to help understand if you=E2=80=99re losing data, = how many have you seen, etc=E2=80=A6. It is a curious number when you = think about how you would want to get and process flow records, = especially when you want to merge, aggregate and or filter the records = to get to an answer to a question. But it is there to help when needed = (debugging, integrity checks when using udp transport, etc =E2=80=A6.). > =20 > When Argus generates a flow record, the output stage puts an Argus = source id and a sequence number in the =E2=80=9Ctrans=E2=80=9D data sub = record. Historically, the trans dsr has been an interesting topic. = Should it used for hop to hop loss detection ? If so then each stage of = an argus data pipeline, source -> radium -> radium -> radium -> disk, = would want to either overwrite the DSR data for its transport, or we = would need to add a trans dsr for each stage of the pipe. Because the = 'seq' is tied to the =E2=80=98srcid=E2=80=99, we have used it to provide = the information needed to recover missing data from the originating = source. > =20 > For data recovery, the idea is that the argus data generator would = have local storage with a small retention time (days), structured as a = standard argus archive, say in 5 minute files, and collectors of the = full stream will use the seqnum to realize that data needs to be = requested from the originator, say when the pipeline is interrupted, to = recover 5 min of data. Or you could be very clever and do a selective = recovery of a full set of missing seqnum=E2=80=99s. > =20 > Now these recovery methods are not in the open source project, but the = seqnum is there to enable anyone to do this in their production network. > =20 > Having any problems ???? > =20 > Carter > =20 >=20 >=20 > On Sep 6, 2019, at 6:49 AM, Giampaolo Bovenzi = <[email protected] <mailto:[email protected]>> wrote: > =20 > Hello everyone, > =20 > I need some clarification on the =E2=80=9CArgus sequence number=E2=80=9D= attribute, because there are no specific explainations in the = documentation. > In particular, I need to understand how it is constructed. > =20 > Thaks for your help, > Giampaolo Bovenzi. > =20 > _______________________________________________ > argus mailing list > [email protected] <mailto:[email protected]> > https://pairlist1.pair.net/mailman/listinfo/argus = <https://pairlist1.pair.net/mailman/listinfo/argus> --Apple-Mail=_55699C12-DB11-4F7D-BFF1-C03743C568FB Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 <html><head><meta http-equiv=3D"Content-Type" content=3D"text/html; = charset=3Dutf-8"></head><body style=3D"word-wrap: break-word; = -webkit-nbsp-mode: space; line-break: after-white-space;" class=3D"">I = think that there maybe some information in knowing the distance between = argus records, but the variables that impact the record distance is = pretty independent of anything that I would think of interest. = Traffic mix, argus configuration, etc =E2=80=A6. And in high = volume traffic sources, the counter rolls over regularly so long term = behavioral trends would be confused by the number, I suspect, but that = can be dealt with, without much issue =E2=80=A6.<div class=3D""><br = class=3D""></div><div class=3D""><div class=3D""><div dir=3D"auto" = style=3D"caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: = normal; text-align: start; text-indent: 0px; text-transform: none; = white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; = line-break: after-white-space;" class=3D""><div dir=3D"auto" = style=3D"caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: = normal; text-align: start; text-indent: 0px; text-transform: none; = white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; = line-break: after-white-space;" class=3D"">I think that some want to use = the sequence number as a quasi key for the transaction, but we have = other transaction ids that are designed for that =E2=80=A6 and of course = the flow key, plus time is a much better discriminator for flow = identification.</div><div dir=3D"auto" style=3D"caret-color: rgb(0, 0, = 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: = none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: = after-white-space;" class=3D""><br class=3D""></div><div dir=3D"auto" = style=3D"caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: = normal; text-align: start; text-indent: 0px; text-transform: none; = white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; = line-break: after-white-space;" class=3D"">Hope all is most = excellent,</div><div dir=3D"auto" style=3D"caret-color: rgb(0, 0, 0); = color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: = none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: = after-white-space;" class=3D"">Carter<br class=3D""> </div></div> </div> <div><br class=3D""><blockquote type=3D"cite" class=3D""><div = class=3D"">On Sep 6, 2019, at 10:47 AM, Giampaolo Bovenzi <<a = href=3D"mailto:[email protected]" = class=3D"">[email protected]</a>> wrote:</div><br = class=3D"Apple-interchange-newline"><div class=3D""><div = class=3D"WordSection1" style=3D"page: WordSection1; caret-color: rgb(0, = 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;"><div style=3D"margin: 0cm 0cm 0.0001pt; = font-size: 11pt; font-family: Calibri, sans-serif;" class=3D"">Hi = Carter,</div><div style=3D"margin: 0cm 0cm 0.0001pt; font-size: 11pt; = font-family: Calibri, sans-serif;" class=3D""><o:p = class=3D""> </o:p></div><div style=3D"margin: 0cm 0cm 0.0001pt; = font-size: 11pt; font-family: Calibri, sans-serif;" class=3D"">thank for = the rapidity and precision of your response.</div><div style=3D"margin: = 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" = class=3D"">I have got no problem with the attribute itself.</div><div = style=3D"margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: = Calibri, sans-serif;" class=3D""><o:p class=3D""> </o:p></div><div = style=3D"margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: = Calibri, sans-serif;" class=3D"">My curiosity comes from using this = attribute to conduct network traffic modeling.</div><div style=3D"margin: = 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" = class=3D"">As I have seen, some researchers are using the Argus seq = number as input to a modeling system, but, given your response, this = attribute introduces a bias and should be discarded to model network = traffic.</div><div style=3D"margin: 0cm 0cm 0.0001pt; font-size: 11pt; = font-family: Calibri, sans-serif;" class=3D""><o:p = class=3D""> </o:p></div><div style=3D"margin: 0cm 0cm 0.0001pt; = font-size: 11pt; font-family: Calibri, sans-serif;" class=3D"">Thanks = again and good work!</div><div style=3D"margin: 0cm 0cm 0.0001pt; = font-size: 11pt; font-family: Calibri, sans-serif;" = class=3D"">GB</div><div style=3D"margin: 0cm 0cm 0.0001pt; font-size: = 11pt; font-family: Calibri, sans-serif;" class=3D""><o:p = class=3D""> </o:p></div><div style=3D"border-style: solid none = none; border-top-width: 1pt; border-top-color: rgb(225, 225, 225); = padding: 3pt 0cm 0cm;" class=3D""><div style=3D"margin: 0cm 0cm = 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif; border: = none; padding: 0cm;" class=3D""><b class=3D"">Da:<span = class=3D"Apple-converted-space"> </span></b><a = href=3D"mailto:[email protected]" style=3D"color: blue; = text-decoration: underline;" class=3D"">[email protected]</a><br = class=3D""><b class=3D"">Inviato:<span = class=3D"Apple-converted-space"> </span></b>venerd=C3=AC 6 = settembre 2019 15:56<br class=3D""><b class=3D"">A:<span = class=3D"Apple-converted-space"> </span></b><a = href=3D"mailto:[email protected]" style=3D"color: blue; = text-decoration: underline;" class=3D"">Argus</a><br class=3D""><b = class=3D"">Oggetto:<span = class=3D"Apple-converted-space"> </span></b>Re: [ARGUS] Attribute = clarification</div></div><div style=3D"margin: 0cm 0cm 0.0001pt; = font-size: 11pt; font-family: Calibri, sans-serif;" class=3D""><o:p = class=3D""> </o:p></div><div style=3D"margin: 0cm 0cm 0.0001pt; = font-size: 11pt; font-family: Calibri, sans-serif;" class=3D"">Hey = Giampaolo,<o:p class=3D""></o:p></div><div class=3D""><div = style=3D"margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: = Calibri, sans-serif;" class=3D"">The Argus sequence number is a = monotonically increasing record number in the ARGUS_TRANSPORT_DSR, the = structure in the argus record that is used to identify the data source = of data during data transport between nodes. The 32-bit int seqnum = is generated by the originating Argus data source, and is used to help = understand if you=E2=80=99re losing data, how many have you seen, = etc=E2=80=A6. It is a curious number when you think about how you = would want to get and process flow records, especially when you want to = merge, aggregate and or filter the records to get to an answer to a = question. But it is there to help when needed (debugging, = integrity checks when using udp transport, etc =E2=80=A6.).<o:p = class=3D""></o:p></div></div><div class=3D""><div style=3D"margin: 0cm = 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" = class=3D""><o:p class=3D""> </o:p></div></div><div class=3D""><div = style=3D"margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: = Calibri, sans-serif;" class=3D"">When Argus generates a flow record, the = output stage puts an Argus source id and a sequence number in the = =E2=80=9Ctrans=E2=80=9D data sub record. Historically, the trans = dsr has been an interesting topic. Should it used for hop to hop = loss detection ? If so then each stage of an argus data pipeline, = source -> radium -> radium -> radium -> disk, would want to = either overwrite the DSR data for its transport, or we would need to add = a trans dsr for each stage of the pipe. Because the 'seq' is tied = to the =E2=80=98srcid=E2=80=99, we have used it to provide the = information needed to recover missing data from the originating = source.<o:p class=3D""></o:p></div></div><div class=3D""><div = style=3D"margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: = Calibri, sans-serif;" class=3D""><o:p = class=3D""> </o:p></div></div><div class=3D""><div style=3D"margin: = 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" = class=3D"">For data recovery, the idea is that the argus data generator = would have local storage with a small retention time (days), structured = as a standard argus archive, say in 5 minute files, and collectors of = the full stream will use the seqnum to realize that data needs to be = requested from the originator, say when the pipeline is interrupted, to = recover 5 min of data. Or you could be very clever and do a selective = recovery of a full set of missing seqnum=E2=80=99s.<o:p = class=3D""></o:p></div></div><div class=3D""><div style=3D"margin: 0cm = 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" = class=3D""><o:p class=3D""> </o:p></div></div><div class=3D""><div = style=3D"margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: = Calibri, sans-serif;" class=3D"">Now these recovery methods are not in = the open source project, but the seqnum is there to enable anyone to do = this in their production network.<o:p class=3D""></o:p></div></div><div = class=3D""><div style=3D"margin: 0cm 0cm 0.0001pt; font-size: 11pt; = font-family: Calibri, sans-serif;" class=3D""><o:p = class=3D""> </o:p></div></div><div class=3D""><div style=3D"margin: = 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" = class=3D"">Having any problems ????<o:p class=3D""></o:p></div></div><div = class=3D""><div style=3D"margin: 0cm 0cm 0.0001pt; font-size: 11pt; = font-family: Calibri, sans-serif;" class=3D""><o:p = class=3D""> </o:p></div></div><div class=3D""><div class=3D""><div = style=3D"margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: = Calibri, sans-serif;" class=3D"">Carter<o:p = class=3D""></o:p></div></div><div class=3D""><div class=3D""><div = style=3D"margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: = Calibri, sans-serif;" class=3D""><o:p = class=3D""> </o:p></div></div><div class=3D""><div style=3D"margin: = 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" = class=3D""><br class=3D""><br class=3D""><o:p = class=3D""></o:p></div><blockquote style=3D"margin-top: 5pt; = margin-bottom: 5pt;" class=3D""><div class=3D""><div style=3D"margin: = 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" = class=3D"">On Sep 6, 2019, at 6:49 AM, Giampaolo Bovenzi <<a = href=3D"mailto:[email protected]" style=3D"color: blue; = text-decoration: underline;" class=3D"">[email protected]</a>>= wrote:<o:p class=3D""></o:p></div></div><div style=3D"margin: 0cm 0cm = 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" = class=3D""><o:p class=3D""> </o:p></div><div class=3D""><div = class=3D""><div style=3D"margin: 0cm 0cm 0.0001pt; font-size: 11pt; = font-family: Calibri, sans-serif;" class=3D"">Hello everyone,<o:p = class=3D""></o:p></div></div><div class=3D""><div style=3D"margin: 0cm = 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" = class=3D""> <o:p class=3D""></o:p></div></div><div class=3D""><div = style=3D"margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: = Calibri, sans-serif;" class=3D"">I need some clarification on the = =E2=80=9CArgus sequence number=E2=80=9D attribute, because there are no = specific explainations in the documentation.<o:p = class=3D""></o:p></div></div><div class=3D""><div style=3D"margin: 0cm = 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" = class=3D"">In particular, I need to understand how it is = constructed.<o:p class=3D""></o:p></div></div><div class=3D""><div = style=3D"margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: = Calibri, sans-serif;" class=3D""> <o:p = class=3D""></o:p></div></div><div class=3D""><div style=3D"margin: 0cm = 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" = class=3D"">Thaks for your help,<o:p class=3D""></o:p></div></div><div = class=3D""><div style=3D"margin: 0cm 0cm 0.0001pt; font-size: 11pt; = font-family: Calibri, sans-serif;" class=3D"">Giampaolo Bovenzi.<o:p = class=3D""></o:p></div></div><div class=3D""><div style=3D"margin: 0cm = 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" = class=3D""> <o:p class=3D""></o:p></div></div><div style=3D"margin: = 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" = class=3D""><span style=3D"font-size: 9pt; font-family: Helvetica, = sans-serif;" class=3D"">_______________________________________________<br= class=3D"">argus mailing list<br class=3D""></span><a = href=3D"mailto:[email protected]" style=3D"color: blue; text-decoration: = underline;" class=3D""><span style=3D"font-size: 9pt; font-family: = Helvetica, sans-serif;" class=3D"">[email protected]</span></a><span = style=3D"font-size: 9pt; font-family: Helvetica, sans-serif;" = class=3D""><br class=3D""></span><a = href=3D"https://pairlist1.pair.net/mailman/listinfo/argus" style=3D"color:= blue; text-decoration: underline;" class=3D""><span style=3D"font-size: = 9pt; font-family: Helvetica, sans-serif;" = class=3D"">https://pairlist1.pair.net/mailman/listinfo/argus</span></a></d= iv></div></blockquote></div></div></div></div></div></blockquote></div><br= class=3D""></div></body></html>= --Apple-Mail=_55699C12-DB11-4F7D-BFF1-C03743C568FB--