Re: Attribute clarification
[email protected] Fri, 6 Sep 2019 09:55:29 -0400
| Newsgroups | gmane.network.argus |
|---|---|
| Message-ID | <[email protected]> |
--Apple-Mail=_106DC5BA-C156-42E3-91D8-9DBA4255079D Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Hey Giampaolo, The Argus sequence number is a monotonically increasing record number in = the ARGUS_TRANSPORT_DSR, the structure in the argus record that is used = to identify the data source of data during data transport between nodes. = The 32-bit int seqnum is generated by the originating Argus data = source, and is used to help understand if you=E2=80=99re losing data, = how many have you seen, etc=E2=80=A6. It is a curious number when you = think about how you would want to get and process flow records, = especially when you want to merge, aggregate and or filter the records = to get to an answer to a question. But it is there to help when needed = (debugging, integrity checks when using udp transport, etc =E2=80=A6.). When Argus generates a flow record, the output stage puts an Argus = source id and a sequence number in the =E2=80=9Ctrans=E2=80=9D data sub = record. Historically, the trans dsr has been an interesting topic. = Should it used for hop to hop loss detection ? If so then each stage of = an argus data pipeline, source -> radium -> radium -> radium -> disk, = would want to either overwrite the DSR data for its transport, or we = would need to add a trans dsr for each stage of the pipe. Because the = 'seq' is tied to the =E2=80=98srcid=E2=80=99, we have used it to provide = the information needed to recover missing data from the originating = source. For data recovery, the idea is that the argus data generator would have = local storage with a small retention time (days), structured as a = standard argus archive, say in 5 minute files, and collectors of the = full stream will use the seqnum to realize that data needs to be = requested from the originator, say when the pipeline is interrupted, to = recover 5 min of data. Or you could be very clever and do a selective = recovery of a full set of missing seqnum=E2=80=99s. Now these recovery methods are not in the open source project, but the = seqnum is there to enable anyone to do this in their production network. Having any problems ???? Carter > On Sep 6, 2019, at 6:49 AM, Giampaolo Bovenzi = <[email protected]> wrote: >=20 > Hello everyone, > =20 > I need some clarification on the =E2=80=9CArgus sequence number=E2=80=9D= attribute, because there are no specific explainations in the = documentation. > In particular, I need to understand how it is constructed. > =20 > Thaks for your help, > Giampaolo Bovenzi. > =20 > _______________________________________________ > argus mailing list > [email protected] <mailto:[email protected]> > https://pairlist1.pair.net/mailman/listinfo/argus = <https://pairlist1.pair.net/mailman/listinfo/argus> --Apple-Mail=_106DC5BA-C156-42E3-91D8-9DBA4255079D Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 <html><head><meta http-equiv=3D"Content-Type" content=3D"text/html; = charset=3Dutf-8"></head><body style=3D"word-wrap: break-word; = -webkit-nbsp-mode: space; line-break: after-white-space;" class=3D"">Hey = Giampaolo,<div class=3D"">The Argus sequence number is a monotonically = increasing record number in the ARGUS_TRANSPORT_DSR, the structure in = the argus record that is used to identify the data source of data during = data transport between nodes. The 32-bit int seqnum is generated = by the originating Argus data source, and is used to help understand if = you=E2=80=99re losing data, how many have you seen, etc=E2=80=A6. = It is a curious number when you think about how you would want to = get and process flow records, especially when you want to merge, = aggregate and or filter the records to get to an answer to a question. = But it is there to help when needed (debugging, integrity checks = when using udp transport, etc =E2=80=A6.).</div><div class=3D""><br = class=3D""></div><div class=3D"">When Argus generates a flow record, the = output stage puts an Argus source id and a sequence number in the = =E2=80=9Ctrans=E2=80=9D data sub record. Historically, the trans = dsr has been an interesting topic. Should it used for hop to hop = loss detection ? If so then each stage of an argus data pipeline, = source -> radium -> radium -> radium -> disk, would want to = either overwrite the DSR data for its transport, or we would need to add = a trans dsr for each stage of the pipe. Because the 'seq' is tied = to the =E2=80=98srcid=E2=80=99, we have used it to provide the = information needed to recover missing data from the originating = source.</div><div class=3D""><br class=3D""></div><div class=3D"">For = data recovery, the idea is that the argus data generator would have = local storage with a small retention time (days), structured as a = standard argus archive, say in 5 minute files, and collectors of the = full stream will use the seqnum to realize that data needs to be = requested from the originator, say when the pipeline is interrupted, to = recover 5 min of data. Or you could be very clever and do a selective = recovery of a full set of missing seqnum=E2=80=99s.</div><div = class=3D""><br class=3D""></div><div class=3D"">Now these recovery = methods are not in the open source project, but the seqnum is there to = enable anyone to do this in their production network.</div><div = class=3D""><br class=3D""></div><div class=3D"">Having any problems = ????</div><div class=3D""><br class=3D""></div><div class=3D""><div = class=3D"">Carter</div><div class=3D""><div class=3D""><br class=3D""> </div> <div><br class=3D""><blockquote type=3D"cite" class=3D""><div = class=3D"">On Sep 6, 2019, at 6:49 AM, Giampaolo Bovenzi <<a = href=3D"mailto:[email protected]" = class=3D"">[email protected]</a>> wrote:</div><br = class=3D"Apple-interchange-newline"><div class=3D""><div = class=3D"WordSection1" style=3D"page: WordSection1; caret-color: rgb(0, = 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;"><div style=3D"margin: 0cm 0cm 0.0001pt; = font-size: 11pt; font-family: Calibri, sans-serif;" class=3D"">Hello = everyone,<o:p class=3D""></o:p></div><div style=3D"margin: 0cm 0cm = 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" = class=3D""><o:p class=3D""> </o:p></div><div style=3D"margin: 0cm = 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" = class=3D"">I need some clarification on the =E2=80=9CArgus sequence = number=E2=80=9D attribute, because there are no specific explainations = in the documentation.<o:p class=3D""></o:p></div><div style=3D"margin: = 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" = class=3D"">In particular, I need to understand how it is = constructed.<o:p class=3D""></o:p></div><div style=3D"margin: 0cm 0cm = 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" = class=3D""><o:p class=3D""> </o:p></div><div style=3D"margin: 0cm = 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" = class=3D"">Thaks for your help,<o:p class=3D""></o:p></div><div = style=3D"margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: = Calibri, sans-serif;" class=3D"">Giampaolo Bovenzi.<o:p = class=3D""></o:p></div><div style=3D"margin: 0cm 0cm 0.0001pt; = font-size: 11pt; font-family: Calibri, sans-serif;" class=3D""><o:p = class=3D""> </o:p></div></div><span style=3D"caret-color: rgb(0, 0, = 0); font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none; float: none; display: inline !important;" = class=3D"">_______________________________________________</span><br = style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: = 12px; font-style: normal; font-variant-caps: normal; font-weight: = normal; letter-spacing: normal; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; text-decoration: none;" class=3D""><span = style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: = 12px; font-style: normal; font-variant-caps: normal; font-weight: = normal; letter-spacing: normal; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; text-decoration: none; float: none; = display: inline !important;" class=3D"">argus mailing list</span><br = style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: = 12px; font-style: normal; font-variant-caps: normal; font-weight: = normal; letter-spacing: normal; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; text-decoration: none;" class=3D""><a = href=3D"mailto:[email protected]" style=3D"font-family: Helvetica; = font-size: 12px; font-style: normal; font-variant-caps: normal; = font-weight: normal; letter-spacing: normal; orphans: auto; text-align: = start; text-indent: 0px; text-transform: none; white-space: normal; = widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; = -webkit-text-stroke-width: 0px;" class=3D"">[email protected]</a><br = style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: = 12px; font-style: normal; font-variant-caps: normal; font-weight: = normal; letter-spacing: normal; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; text-decoration: none;" class=3D""><a = href=3D"https://pairlist1.pair.net/mailman/listinfo/argus" = style=3D"font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = orphans: auto; text-align: start; text-indent: 0px; text-transform: = none; white-space: normal; widows: auto; word-spacing: 0px; = -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" = class=3D"">https://pairlist1.pair.net/mailman/listinfo/argus</a></div></bl= ockquote></div><br class=3D""></div></div></body></html>= --Apple-Mail=_106DC5BA-C156-42E3-91D8-9DBA4255079D--