Re: Cisco 2970 and VIA Rhine PXE bootstrap problems

On Wed, Mar 15, 2006 at 11:31:35PM +1000, Ian Latter wrote:
> Hello Tony,
> 
> 
>   I haven't seen this before, but ...
> 
>   If, all things being equal, your cluster works when
> it has hosts with static IP assignments, then you 
> might find that it's the port negotiation between the 
> NIC and the Switch that is causing your problem.  In 
> *just* the right circumstances, you can have some 
> switches spend minutes negotiating link-up 
> information (like Duplex and Rate).  If this negotiation
> is running longer than your PXE/BootP time-out then
> this would be bad.
> 
>   Try setting your link speed and rate in the switch for
> the ports used by static hosts (those nodes that 
> aren't often physically relocated) - removing auto-
> negotiation.

 I haven't had autoneg problem within my clusters (My D-link gigabit switch
has served well :), but the University network uses Cisco switches,
sometimes with autonegotiation disabled.  The only problems I've ever had
with autoneg, ironically, are with Cisco switches where some genius turned off
autonegotiation, presumably in the interests of avoiding problems!  I
have to have   mii-tool -F 100baseTx-FD  in my network config stuff on my
desktop and on the cluster master node (for its external connection).

 (On my desktop, since I want to run a DHCP client (dhcp3-client), I had to
add a script /etc/dhcp3/dhclient-exit-hooks.d/local-force-100tx-fd:

---
# 2k6/1/16: Peter Cordes
# handle setting media, which is needed because 
# dhclient brings up the interface, which restarts autonegotiation.

# this is needed in the Chase building because autoneg is unreliable, or at
# least very slow, with sis900

# The NIC still gets reset after PREINIT, so this isn't the way to do it.
#if [ x$reason = xPREINIT ]; then
#   mii-tool -A 100baseTx-FD $interface
#   echo "$0: forcing 100baseTx, full duplex on $interface"
#   mii-tool -F 100baseTx-FD $interface
#fi

# have to set   media "100baseTx-FD";  in dhclient.conf
if [ x$reason = xMEDIUM ]; then
   echo "$0: forcing medium $medium on $interface"
   mii-tool -F $medium $interface
fi
---

 As you can see, it doesn't actually just force 100baseTx-FD, it responds to
a  media   setting in dhclient.conf.

 BTW, this is on Debian.

 This has actually been working for me for the last couple months, and I
never have to manually do anything after booting up my desktop at work to
get networking working.  This is in contrast to the few years when I've
mostly just left my computer on, partly for bittorrent (KNOPPIX and things
like that, nothing too inappropriate for work!), partly because broken
networking was a pain.

 If Cisco had a setting that would make the switch port use the autoneg
protocol, but only offer a single speed/duplex that you wanted to force,
that would totally take care of the problem; people with well-behaved
network cards wouldn't be forced to manually configure anything on their
end.  (I read plenty of harsh words directed at Cisco while googling about
autonegotiation...  I haven't personally used Cisco gear, just been on the
receiving end of its apparent braindeath.)  I think turning off autoneg
precluded the use of flow control, since it's autonegotiated and mii-tool
doesn't have a way to force it on that I know of.

mycluster:~# mii-tool 
eth0: negotiated 100baseTx-FD flow-control, link ok
eth1: 100 Mbit, full duplex, link ok

 The moral of this story is:  autonegotiation is good.  All modern NICs do
it correctly and compatibly (right?).  Leave it on and you won't have to
manually do anything to every machine whose default isn't 100baseTx-FD.

-- 
#define X(x,y) x##y
Peter Cordes ;  e-mail: X(peter@cor , des.ca)

"The gods confound the man who first found out how to distinguish the hours!
 Confound him, too, who in this place set up a sundial, to cut and hack
 my day so wretchedly into small pieces!" -- Plautus, 200 BC


-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642