Re: Malfunctioning of argus -S option for some pcaps
Hang Guo <[email protected]> Wed, 3 Jul 2019 14:17:41 -0700
| Newsgroups | gmane.network.argus |
|---|---|
| Message-ID | <CAOPGVOeR1VEfPtoj52aqVeXuwgYt3=sjZ5mVJ0=KbWT=rNcp+A@mail.gmail.com> |
--000000000000ce5d51058ccd65bb Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable I see. Argus uses this single queue to simulate having a seperate timer for each flow. Thanks for the great explanation Carter! -Hang On Wed, Jul 3, 2019 at 7:06 AM <[email protected]> wrote: > Hey Hang, > Each flow cache that is being tracked is placed in a single "status=E2=80= =9D > queue, in start time order. The end of the queue is used to manage the > status interval. The output processor just looks at the end of this que= ue > to decide if it's time to report the flow cache's values. This saves > having to manage a timer for each flow cache, which can number in the > millions. > > Of course, each flow cache entry has an independent timestamp, the first > packet=E2=80=99s timestamp, which is the =E2=80=9Cstart time". This work= s, because argus, > as a real-time flow sensor, reads packets off the wire, where all the > packets MUST come in in time order (nature of networking a time stamping > each packet as it comes in, using the same clock). When argus reads > packets from a file, or a stream of packets from a distributed packet > broker, it gets a little weird, as the arrival of packets are not > real-time, and can be delayed, shaped and / or reordered. > > To deal with the weirdness, argus uses a very specific strategy. When no= t > reading realtime packets using a real-time sensor, you have to decide if > you want to =E2=80=9Cbelieve" the timestamp in the packet or do you inste= ad use the > realtime system clock when argus receives the packet (whenever that might > be). Argus uses the timestamp provided by the packet capture facility as > the realtime clock, and plays a lot of games trying to figure out what to > do next when there are gaps in time. The reasons are huge, from a > practical perspective, as you want deterministic results when processing > packets. The only requirement is that the packets come in mostly in time > order. > > Carter > [image: QoSient] > <https://urldefense.proofpoint.com/v2/url?u=3Dhttp-3A__qosient.com&d=3DDw= MFaQ&c=3DclK7kQUTWtAVEOVIgvi0NU5BOUHhpN0H8p7CSfnc_gI&r=3DGkcNqHVldHLLhP7B8K= __Og&m=3DuOYAcJz5kWdwAlN2DkpHXdW1nhIwo7QmA_wgvPT6JV0&s=3DDFLtrPSbsLRZvb0pYx= 7iRl0e13IlVODQh3zRsEoPb_Q&e=3D> > > Carter Bullard <[email protected]>=E2=80=A2 Chairman / Founder / CTO > 150 E 57th Street, Suite 12D > New York, New York 10022-2795 > Phone +1.212.588.9133 =E2=80=A2 Mobile +1.917.497.9494 > > > > On Jul 2, 2019, at 10:48 PM, Hang Guo <[email protected]> wrote: > > Thanks Carter, that saves life!!!! > > Could you help me understand one more bit about -S option: when implemtin= g > this report interval (-S), I guess argus uses one timer for all flows > instead of a seperate timer for each flow right? > > Thanks, > -Hang > > On Tue, Jul 2, 2019 at 6:28 PM <[email protected]> wrote: > >> Hey Hang, >> The problem is that your packets are not in time order, so argus is doin= g >> the right thing. Best example is around packet # 3848, where the packet >> timestamps jumps back 6.5 hours. Nothing argus can do with that but tal= ly >> the packets =E2=80=A6 and all the flows will be messed up for a little w= hile, until >> the packet timestamps move ahead in front of the largest startime. If y= ou >> can get the packets sorted in time, then things should work fine. >> >> Carter >> >> On Jul 2, 2019, at 4:19 PM, Hang Guo <[email protected]> wrote: >> >> Hi, >> >> I found argus -S option malfunctioning for some pcaps. For example, when >> running argus -S 10 with the pcap attached (MAC and IP anamoyzed, payloa= d >> dropped for privacy), instead of reportting every 5-tuple flows every 10 >> seconds, duration of some reported 5-tuple flows (as pasted below) are >> hundreds of seconds. Just wonder what is the possible cause and is there= a >> fix to this? >> >> argus -S 10 -r dur_test_anon.pcapng -w - | ra -c "," -r - -s dur | sort >>> -nr | head -5 >>> 4865.550781 >>> 296.601562 >>> 296.393066 >>> 294.411255 >>> 292.840790 >> >> >> Thanks, >> -Hang >> <dur_test_anon.pcapng> >> >> >> > --000000000000ce5d51058ccd65bb Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr">I see. Argus uses this single queue to simulate having a s= eperate=C2=A0timer for each flow.=C2=A0 Thanks for the great explanation Ca= rter!=C2=A0<div><br></div><div>-Hang</div></div><br><div class=3D"gmail_quo= te"><div dir=3D"ltr" class=3D"gmail_attr">On Wed, Jul 3, 2019 at 7:06 AM &l= t;<a href=3D"mailto:[email protected]" target=3D"_blank">[email protected]= m</a>> wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"margin= :0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"= ><div>Hey Hang,<div>Each flow cache that is being tracked is placed in a si= ngle "status=E2=80=9D queue, in start time order.=C2=A0 The end of the= queue is used to manage the status interval. =C2=A0 The output processor j= ust looks at the end of this queue to decide if it's time to report the= flow cache's values.=C2=A0 This saves having to manage a timer for eac= h flow cache, which can number in the millions.</div><div><br></div><div>Of= course, each flow cache entry has an independent timestamp, the first pack= et=E2=80=99s timestamp, which is the =E2=80=9Cstart time".=C2=A0 This = works, because argus, as a real-time flow sensor, reads packets off the wir= e, where all the packets MUST come in in time order (nature of networking a= time stamping each packet as it comes in, using the same clock).=C2=A0 Whe= n argus reads packets from a file, or a stream of packets from a distribute= d packet broker, it gets a little weird, as the arrival of packets are not = real-time, and can be delayed, shaped and / or reordered. =C2=A0</div><div>= <br></div><div>To deal with the weirdness, argus uses a very specific strat= egy.=C2=A0 When not reading realtime packets using a real-time sensor, you = have to decide if you want to =E2=80=9Cbelieve" the timestamp in the p= acket or do you instead use the realtime system clock when argus receives t= he packet (whenever that might be).=C2=A0 Argus uses the timestamp provided= by the packet capture facility as the realtime clock, and plays a lot of g= ames trying to figure out what to do next when there are gaps in time.=C2= =A0 The reasons are huge, from a practical perspective, as you want determi= nistic results when processing packets.=C2=A0 The only requirement is that = the packets come in mostly in time order.</div><div><br></div><div><div>Car= ter<br><div> <div dir=3D"auto" style=3D"color:rgb(0,0,0);letter-spacing:normal;text-alig= n:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing= :0px;text-decoration:none"><div dir=3D"auto" style=3D"color:rgb(0,0,0);lett= er-spacing:normal;text-align:start;text-indent:0px;text-transform:none;whit= e-space:normal;word-spacing:0px;text-decoration:none"><table id=3D"gmail-m_= -4388168953611998279gmail-m_-8577873300417275062sig" width=3D"360" cellspac= ing=3D"0" cellpadding=3D"0" style=3D"width:360px;margin:0px;padding:0px"><t= body><tr><td width=3D"142" style=3D"width:142px;margin:0px;padding:0px"><a = href=3D"https://urldefense.proofpoint.com/v2/url?u=3Dhttp-3A__qosient.com&a= mp;d=3DDwMFaQ&c=3DclK7kQUTWtAVEOVIgvi0NU5BOUHhpN0H8p7CSfnc_gI&r=3DG= kcNqHVldHLLhP7B8K__Og&m=3DuOYAcJz5kWdwAlN2DkpHXdW1nhIwo7QmA_wgvPT6JV0&a= mp;s=3DDFLtrPSbsLRZvb0pYx7iRl0e13IlVODQh3zRsEoPb_Q&e=3D" title=3D"QoSie= nt" style=3D"border:none;text-decoration:none" target=3D"_blank"><img alt= =3D"QoSient" style=3D"border: none; height: 48px; font-family: "Gill S= ans", sans-serif; font-weight: 100; font-size: 48px; line-height: 48px= ; color: rgb(0, 0, 0);"><span class=3D"gmail-m_-4388168953611998279gmail-m_= -8577873300417275062Apple-converted-space">=C2=A0</span><span class=3D"gmai= l-m_-4388168953611998279gmail-m_-8577873300417275062Apple-converted-space">= =C2=A0</span><span class=3D"gmail-m_-4388168953611998279gmail-m_-8577873300= 417275062Apple-converted-space">=C2=A0</span><span class=3D"gmail-m_-438816= 8953611998279gmail-m_-8577873300417275062Apple-converted-space">=C2=A0</spa= n><span class=3D"gmail-m_-4388168953611998279gmail-m_-8577873300417275062Ap= ple-converted-space">=C2=A0</span><span class=3D"gmail-m_-43881689536119982= 79gmail-m_-8577873300417275062Apple-converted-space">=C2=A0</span></a><span= class=3D"gmail-m_-4388168953611998279gmail-m_-8577873300417275062Apple-con= verted-space">=C2=A0</span><span class=3D"gmail-m_-4388168953611998279gmail= -m_-8577873300417275062Apple-converted-space">=C2=A0</span><span class=3D"g= mail-m_-4388168953611998279gmail-m_-8577873300417275062Apple-converted-spac= e">=C2=A0</span><span class=3D"gmail-m_-4388168953611998279gmail-m_-8577873= 300417275062Apple-converted-space">=C2=A0</span></td><td width=3D"20" style= =3D"width:16px;min-width:16px;max-width:16px;margin:0px;padding:0px">=C2=A0= </td><td style=3D"margin:0px;padding:0px;border-collapse:collapse"><table i= d=3D"gmail-m_-4388168953611998279gmail-m_-8577873300417275062sig2" cellspac= ing=3D"0" cellpadding=3D"0"><tbody><tr><td style=3D"font-family:"Gill = Sans Light",sans-serif;white-space:nowrap"><a href=3D"mailto:carter@qo= sient.com" style=3D"border:none;text-decoration:none" target=3D"_blank"><sp= an style=3D"font-size:16px;line-height:16px;color:rgb(0,0,176)">Carter Bull= ard</span><span class=3D"gmail-m_-4388168953611998279gmail-m_-8577873300417= 275062Apple-converted-space">=C2=A0</span><span class=3D"gmail-m_-438816895= 3611998279gmail-m_-8577873300417275062Apple-converted-space">=C2=A0</span><= /a><span style=3D"font-size:12px;line-height:14px;color:rgb(0,0,0)">=E2=80= =A2</span><span class=3D"gmail-m_-4388168953611998279gmail-m_-8577873300417= 275062Apple-converted-space">=C2=A0</span><span class=3D"gmail-m_-438816895= 3611998279gmail-m_-8577873300417275062Apple-converted-space">=C2=A0</span><= span style=3D"font-size:12px;line-height:14px;color:rgb(0,0,0)">Chairman / = Founder / CTO</span></td></tr><tr><td style=3D"font-family:"Gill Sans = Light",sans-serif;white-space:nowrap"><span style=3D"font-size:12px;li= ne-height:14px;color:rgb(0,0,0)">150 E 57th Street, Suite 12D</span></td></= tr><tr><td style=3D"font-family:"Gill Sans Light",sans-serif;whit= e-space:nowrap"><span style=3D"font-size:12px;line-height:14px;color:rgb(0,= 0,0)">New York, New York 10022-2795</span></td></tr><tr><td style=3D"font-f= amily:"Gill Sans Light",sans-serif;white-space:nowrap"><span styl= e=3D"font-size:12px;line-height:12px;color:rgb(0,0,0)">Phone +1.212.588.913= 3 =E2=80=A2 Mobile +1.917.497.9494</span></td></tr></tbody></table></td></t= r><tr></tr></tbody></table><br>=C2=A0</div></div> </div> <div><br><blockquote type=3D"cite"><div>On Jul 2, 2019, at 10:48 PM, Hang G= uo <<a href=3D"mailto:[email protected]" target=3D"_blank">[email protected]= </a>> wrote:</div><br class=3D"gmail-m_-4388168953611998279gmail-m_-8577= 873300417275062Apple-interchange-newline"><div><div dir=3D"ltr">Thanks Cart= er, that saves life!!!!<div><br></div><div>Could you help me understand one= more bit about -S option: when implemting this report interval (-S), I gue= ss argus uses one timer for all flows instead of a seperate timer for each = flow right?</div><div><br></div><div>Thanks,</div><div>-Hang=C2=A0</div></d= iv><br><div class=3D"gmail_quote"><div dir=3D"ltr" class=3D"gmail_attr">On = Tue, Jul 2, 2019 at 6:28 PM <<a href=3D"mailto:[email protected]" targe= t=3D"_blank">[email protected]</a>> wrote:<br></div><blockquote class= =3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rg= b(204,204,204);padding-left:1ex"><div>Hey Hang,<div>The problem is that you= r packets are not in time order, so argus is doing the right thing.=C2=A0 B= est example is around packet # 3848, where the packet timestamps jumps back= 6.5 hours.=C2=A0 Nothing argus can do with that but tally the packets =E2= =80=A6 and all the flows will be messed up for a little while, until the pa= cket timestamps move ahead in front of the largest startime.=C2=A0 If you c= an get the packets sorted in time, then things should work fine.</div><div>= <br><div>Carter<br> <div><br><blockquote type=3D"cite"><div>On Jul 2, 2019, at 4:19 PM, Hang Gu= o <<a href=3D"mailto:[email protected]" target=3D"_blank">[email protected]<= /a>> wrote:</div><br class=3D"gmail-m_-4388168953611998279gmail-m_-85778= 73300417275062gmail-m_4501021760447724577Apple-interchange-newline"><div><d= iv dir=3D"ltr">Hi,<div><br></div><div><div>I found argus -S option malfunct= ioning for some pcaps. For example, when running argus -S 10 with the pcap = attached (MAC and IP anamoyzed, payload dropped for privacy), instead of re= portting every 5-tuple flows every 10 seconds, duration of some reported 5-= tuple flows (as pasted below) are hundreds of seconds. Just wonder what is = the possible cause and is there a fix to this?</div><div><br></div><blockqu= ote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px= solid rgb(204,204,204);padding-left:1ex">argus -S 10 -r dur_test_anon.pcap= ng -w - | ra -c "," -r - -s dur | sort -nr | head -5<br>4865.5507= 81<br>296.601562<br>296.393066<br>294.411255<br>292.840790</blockquote><div= ><br></div><div>Thanks,</div><div>-Hang=C2=A0</div></div></div> <span id=3D"gmail-m_-4388168953611998279gmail-m_-8577873300417275062gmail-m= _4501021760447724577cid:f_jxm9ab750"><dur_test_anon.pcapng></span></d= iv></blockquote></div><br></div></div></div></blockquote></div> </div></blockquote></div><br></div></div></div></blockquote></div> --000000000000ce5d51058ccd65bb--