Re: [DLC]help in investigating a possible packet storm

You might also want to enable spanning tree on the switches so that
they'll automatically detect and shut down loops.  Keep in mind that
some applications that require the network to pass traffic immediately
upon linkup (e.g. PXE, NetBoot) will probably need spanning tree
disabled or put in "portfast" mode on endpoint ports, but if you're not
using them it's not going to adversely affect the network.

Ryan Riske


Politik Durden wrote:
> Thanks all for the replies and good advice. Two loops and a shady
> coupler (on an uplink line of all places) were the culprit. Pulled some
> logs and found out that the overnight scripts had been steadily
> increasing in run time from an hour to almost 8 hours over the past 2 or
> 3 months. It's so appropriate that these issues all came to a head at
> closing time on April Fools Day.
> 
> We're running Wireshark (thanks John) and virus scans (thanks Greg)
> overnight but traffic is normal......for now. They're also going to
> update the firmware on all switches this weekend.
> 
> Thanks again :-)
> 
> --- On *Thu, 4/1/10, Politik Durden /<politikdurden-/[email protected]>/* wrote:
> 
> 
>     From: Politik Durden <politikdurden-/[email protected]>
>     Subject: [DLC]help in investigating a possible packet storm
>     To: "Depaul Linux" <[email protected]>, "UFO Mail list"
>     <ufo-Y4FUXIRBYHQvUt+Q7UkouvU/[email protected]>
>     Date: Thursday, April 1, 2010, 11:15 PM
> 
>     Hello all,
> 
>     Going to a client site at 6 AM tomorrow because at about 5 PM today
>     (Thursday) all network traffic started getting really really slow.
> 
>     Here's what I know:
> 
>     - no recent changes (no new switch, NIC, changes to static routes,
>     config changes, patches/upgrades, etc)
> 
>     - about a dozen switches feed into a 3COM switch (no model #s yet).
>     ballpark of 2 to 3 hundred nodes total
> 
>     - no protocols are used, all devices are in "dumb" mode and act as
>     just a plain 'ol switch. some can be managed but no features (snmp,
>     etc) are turned on.
> 
>     - most nodes *seem* to be pingable from both sides of the firewall,
>     but everything is just crawling.
> 
>     - nothing (reports, scripts, etc) is timing out, but everything is
>     just super super slow.
> 
>     They tried swapping out switches one at a time to narrow down the
>     culprit and that helped for a bit, but then traffic slowed down
>     again and they couldn't really do any more during production hours.
> 
>     Theories:
> 
>     - Can one bad port cause this kind of a traffic jam ? They started
>     diags on all the major nodes (server NICs, the central 3COM switch,
>     etc) but nothing obvious so far.
> 
>     - Some sort of protocol/feature was turned on by mistake and now all
>     the switches are confused ? A quick "topeka" (ha!!) points to
>     stories of spanning tree causing these kinds of traffic jams.
> 
>     - Somehow a loop got introduced ?
> 
>     What I really need is suggestions on a good free traffic tool,
>     something we can install on two or three laptops and put each switch
>     through its paces. Any ideas ?
> 
>     Thanks in advance for your comments. This lot always points me in
>     the right direction :-)
> 
> 
>     _______________________________________________
>     DLC mailing list
>     [email protected]
>     http://mailman.depaul.edu/mailman/listinfo/dlc
>     Use the Above Link to Unsubcribe!!
>     http://linux.depaul.edu/
> 
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> DLC mailing list
> [email protected]
> http://mailman.depaul.edu/mailman/listinfo/dlc
> Use the Above Link to Unsubcribe!!
> http://linux.depaul.edu/
_______________________________________________
DLC mailing list
[email protected]
http://mailman.depaul.edu/mailman/listinfo/dlc
Use the Above Link to Unsubcribe!!
http://linux.depaul.edu/