R: Attribute clarification
Giampaolo Bovenzi <[email protected]> Fri, 6 Sep 2019 16:47:43 +0200
| Newsgroups | gmane.network.argus |
|---|---|
| Message-ID | <[email protected]> |
--_C7453E22-8994-4A0E-B3D4-A65D23ECA203_ Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Hi Carter, thank for the rapidity and precision of your response. I have got no problem with the attribute itself. My curiosity comes from using this attribute to conduct network traffic mod= eling. As I have seen, some researchers are using the Argus seq number as input to= a modeling system, but, given your response, this attribute introduces a b= ias and should be discarded to model network traffic. Thanks again and good work! GB Da: [email protected] Inviato: venerd=C3=AC 6 settembre 2019 15:56 A: Argus Oggetto: Re: [ARGUS] Attribute clarification Hey Giampaolo, The Argus sequence number is a monotonically increasing record number in th= e ARGUS_TRANSPORT_DSR, the structure in the argus record that is used to id= entify the data source of data during data transport between nodes. =C2=A0T= he 32-bit int seqnum is generated by the originating Argus data source, and= is used to help understand if you=E2=80=99re losing data, how many have yo= u seen, etc=E2=80=A6. =C2=A0It is a curious number when you think about how= you would want to get and process flow records, especially when you want t= o merge, aggregate and or filter the records to get to an answer to a quest= ion. =C2=A0But it is there to help when needed (debugging, integrity checks= when using udp transport, etc =E2=80=A6.). When Argus generates a flow record, the output stage puts an Argus source i= d and a sequence number in the =E2=80=9Ctrans=E2=80=9D data sub record. =C2= =A0Historically, the trans dsr has been an interesting topic. =C2=A0Should = it used for hop to hop loss detection ? =C2=A0If so then each stage of an a= rgus data pipeline, source -> radium -> radium -> radium -> disk, would wan= t to either overwrite the DSR data for its transport, or we would need to a= dd a trans dsr for each stage of the pipe. =C2=A0 Because the 'seq' is tied= to the =E2=80=98srcid=E2=80=99, we have used it to provide the information= needed to recover missing data from the originating source. For data recovery, the idea is that the argus data generator would have loc= al storage with a small retention time (days), structured as a standard arg= us archive, say in 5 minute files, and collectors of the full stream will u= se the seqnum to realize that data needs to be requested from the originato= r, say when the pipeline is interrupted, to recover 5 min of data. Or you c= ould be very clever and do a selective recovery of a full set of missing se= qnum=E2=80=99s. Now these recovery methods are not in the open source project, but the seqn= um is there to enable anyone to do this in their production network. Having any problems ???? Carter On Sep 6, 2019, at 6:49 AM, Giampaolo Bovenzi <[email protected]> = wrote: Hello everyone, =C2=A0 I need some clarification on the =E2=80=9CArgus sequence number=E2=80=9D at= tribute, because there are no specific explainations in the documentation. In particular, I need to understand how it is constructed. =C2=A0 Thaks for your help, Giampaolo Bovenzi. =C2=A0 _______________________________________________ argus mailing list [email protected] https://pairlist1.pair.net/mailman/listinfo/argus --_C7453E22-8994-4A0E-B3D4-A65D23ECA203_ Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset="utf-8" <html xmlns:o=3D"urn:schemas-microsoft-com:office:office" xmlns:w=3D"urn:sc= hemas-microsoft-com:office:word" xmlns:m=3D"http://schemas.microsoft.com/of= fice/2004/12/omml" xmlns=3D"http://www.w3.org/TR/REC-html40"><head><meta ht= tp-equiv=3DContent-Type content=3D"text/html; charset=3Dutf-8"><meta name= =3DGenerator content=3D"Microsoft Word 15 (filtered medium)"><style><!-- /* Font Definitions */ @font-face {font-family:Helvetica; panose-1:2 11 5 4 2 2 2 2 2 4;} @font-face {font-family:"Cambria Math"; panose-1:2 4 5 3 5 4 6 3 2 4;} @font-face {font-family:Calibri; panose-1:2 15 5 2 2 2 4 3 2 4;} /* Style Definitions */ p.MsoNormal, li.MsoNormal, div.MsoNormal {margin:0cm; margin-bottom:.0001pt; font-size:11.0pt; font-family:"Calibri",sans-serif;} a:link, span.MsoHyperlink {mso-style-priority:99; color:blue; text-decoration:underline;} .MsoChpDefault {mso-style-type:export-only;} @page WordSection1 {size:612.0pt 792.0pt; margin:70.85pt 2.0cm 2.0cm 2.0cm;} div.WordSection1 {page:WordSection1;} --></style></head><body lang=3DIT link=3Dblue vlink=3D"#954F72"><div class= =3DWordSection1><p class=3DMsoNormal>Hi Carter,</p><p class=3DMsoNormal><o:= p> </o:p></p><p class=3DMsoNormal>thank for the rapidity and precision= of your response.</p><p class=3DMsoNormal>I have got no problem with the a= ttribute itself.</p><p class=3DMsoNormal><o:p> </o:p></p><p class=3DMs= oNormal>My curiosity comes from using this attribute to conduct network tra= ffic modeling.</p><p class=3DMsoNormal>As I have seen, some researchers are= using the Argus seq number as input to a modeling system, but, given your = response, this attribute introduces a bias and should be discarded to model= network traffic.</p><p class=3DMsoNormal><o:p> </o:p></p><p class=3DM= soNormal>Thanks again and good work!</p><p class=3DMsoNormal>GB</p><p class= =3DMsoNormal><o:p> </o:p></p><div style=3D'mso-element:para-border-div= ;border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm'><p c= lass=3DMsoNormal style=3D'border:none;padding:0cm'><b>Da: </b><a href=3D"ma= ilto:[email protected]">[email protected]</a><br><b>Inviato: </b>venerd= =C3=AC 6 settembre 2019 15:56<br><b>A: </b><a href=3D"mailto:argus-info@lis= ts.andrew.cmu.edu">Argus</a><br><b>Oggetto: </b>Re: [ARGUS] Attribute clari= fication</p></div><p class=3DMsoNormal><o:p> </o:p></p><p class=3DMsoN= ormal>Hey Giampaolo,<o:p></o:p></p><div><p class=3DMsoNormal>The Argus sequ= ence number is a monotonically increasing record number in the ARGUS_TRANSP= ORT_DSR, the structure in the argus record that is used to identify the dat= a source of data during data transport between nodes. The 32-bit int = seqnum is generated by the originating Argus data source, and is used to he= lp understand if you=E2=80=99re losing data, how many have you seen, etc=E2= =80=A6. It is a curious number when you think about how you would wan= t to get and process flow records, especially when you want to merge, aggre= gate and or filter the records to get to an answer to a question. But= it is there to help when needed (debugging, integrity checks when using ud= p transport, etc =E2=80=A6.).<o:p></o:p></p></div><div><p class=3DMsoNormal= ><o:p> </o:p></p></div><div><p class=3DMsoNormal>When Argus generates = a flow record, the output stage puts an Argus source id and a sequence numb= er in the =E2=80=9Ctrans=E2=80=9D data sub record. Historically, the = trans dsr has been an interesting topic. Should it used for hop to ho= p loss detection ? If so then each stage of an argus data pipeline, s= ource -> radium -> radium -> radium -> disk, would want to eith= er overwrite the DSR data for its transport, or we would need to add a tran= s dsr for each stage of the pipe. Because the 'seq' is tied to the = =E2=80=98srcid=E2=80=99, we have used it to provide the information needed = to recover missing data from the originating source.<o:p></o:p></p></div><d= iv><p class=3DMsoNormal><o:p> </o:p></p></div><div><p class=3DMsoNorma= l>For data recovery, the idea is that the argus data generator would have l= ocal storage with a small retention time (days), structured as a standard a= rgus archive, say in 5 minute files, and collectors of the full stream will= use the seqnum to realize that data needs to be requested from the origina= tor, say when the pipeline is interrupted, to recover 5 min of data. Or you= could be very clever and do a selective recovery of a full set of missing = seqnum=E2=80=99s.<o:p></o:p></p></div><div><p class=3DMsoNormal><o:p> = </o:p></p></div><div><p class=3DMsoNormal>Now these recovery methods are no= t in the open source project, but the seqnum is there to enable anyone to d= o this in their production network.<o:p></o:p></p></div><div><p class=3DMso= Normal><o:p> </o:p></p></div><div><p class=3DMsoNormal>Having any prob= lems ????<o:p></o:p></p></div><div><p class=3DMsoNormal><o:p> </o:p></= p></div><div><div><p class=3DMsoNormal>Carter<o:p></o:p></p></div><div><div= ><p class=3DMsoNormal><o:p> </o:p></p></div><div><p class=3DMsoNormal>= <br><br><o:p></o:p></p><blockquote style=3D'margin-top:5.0pt;margin-bottom:= 5.0pt'><div><p class=3DMsoNormal>On Sep 6, 2019, at 6:49 AM, Giampaolo Bove= nzi <<a href=3D"mailto:[email protected]">giampaolo.bovenzi@uni= na.it</a>> wrote:<o:p></o:p></p></div><p class=3DMsoNormal><o:p> </= o:p></p><div><div><p class=3DMsoNormal>Hello everyone,<o:p></o:p></p></div>= <div><p class=3DMsoNormal> <o:p></o:p></p></div><div><p class=3DMsoNor= mal>I need some clarification on the =E2=80=9CArgus sequence number=E2=80= =9D attribute, because there are no specific explainations in the documenta= tion.<o:p></o:p></p></div><div><p class=3DMsoNormal>In particular, I need t= o understand how it is constructed.<o:p></o:p></p></div><div><p class=3DMso= Normal> <o:p></o:p></p></div><div><p class=3DMsoNormal>Thaks for your = help,<o:p></o:p></p></div><div><p class=3DMsoNormal>Giampaolo Bovenzi.<o:p>= </o:p></p></div><div><p class=3DMsoNormal> <o:p></o:p></p></div><p cla= ss=3DMsoNormal><span style=3D'font-size:9.0pt;font-family:"Helvetica",sans-= serif'>_______________________________________________<br>argus mailing lis= t<br></span><a href=3D"mailto:[email protected]"><span style=3D'font-size:9= .0pt;font-family:"Helvetica",sans-serif'>[email protected]</span></a><span = style=3D'font-size:9.0pt;font-family:"Helvetica",sans-serif'><br></span><a = href=3D"https://pairlist1.pair.net/mailman/listinfo/argus"><span style=3D'f= ont-size:9.0pt;font-family:"Helvetica",sans-serif'>https://pairlist1.pair.n= et/mailman/listinfo/argus</span></a><o:p></o:p></p></div></blockquote></div= ></div></div><p class=3DMsoNormal><o:p> </o:p></p><p class=3DMsoNormal= ><o:p> </o:p></p></div></body></html>= --_C7453E22-8994-4A0E-B3D4-A65D23ECA203_--