Re: [myrinet] static mapping?

;
; Use this just to map the network once, and write a route file
; to be used for future static mapping, without probing.  Outputs
; are:
;   static.map
;   static.routes
;   static.hosts
;
; --pw, 8/30/00
;

;active args file for gm mapper.
;options are disabled by commenting them out with semicolons

;first we have a section for the mapper module. This section begins
;with the token -mapper-args.

-mapper-args




;-map-file is the name of a file the mapper will periodically create
;showing the network topology. This file can be displayed with the
;merlin java program in the gm/mt/merlin directory. Also this file can
;be used as input for the static_mapper and the file_mapper, two
;mappers that configure a myrinet without actually mapping it. If this
;option is disabled then no map files will be made. If the filename is
;not a full pathname, then the file will be created in the directory
;the mapper was started from. In the evil case of Microsoft Windows NT
;(tm) the map file might end up in the /WINNT/GM directory.

-map-file static.map



;-counters-file is the name of a file the mapper will periodically
;create listing various mapper counters and their values. You can look
;at this file to see how many map messages the mapper has sent out and
;so on. The details about full pathnames and evil operating systems are
;the same as for the -map-file option (above). You can use the java
;program mt/watch to watch the ever changing counters file

;-counters-file mapper.counters



;-host-file is the name of a database of Myrinet board IDs, network
;ostnames and GM IDs. It is important for GM that once a node is
;assigned a GM ID, that assignement never changes. To remember the
;assignment between runs, the mapper uses the hosts-file. There are
;many other confusing parts to solving the consistent GM ID assignment
;problem. The -make-hosts option is related.

-host-file static.hosts



;override host table filename to be machine_name.hosts. This is a
;strange option.
;-unique-host-file



;-routes-file is the name of a file the mapper will write a complete
;set of network routes to (if enabled). This file can be used along
;with a map file as -input to the file_mapper. You can used the java
;program merlin in gm/mt/merlin to look at these routes superimposed
;on a diagram of the network. You can also read the file yourself, it
;is a simple text file, like all files the mapper makes.

-routes-file static.routes



;this option is for the static_mapper and the file_mapper. It tells
;these mappers which host in a map file the mapper should correspond
;to.

;-mapper-name avio1



;options for configuring cspi nodes on power up.
;on the first round only configure this host 
;-boot-host 0060dd7fee3d 
; fix for 2610 not accepting single route config messages w/ -boot-host
;-fish-guts


;-map-once tells the mapper to terminate after it has mapped the
;-network and configured it. -unchanged-maps is like -map-once only
;the mapper terminates after a certain number of mapping cycles in
;which the map doesn't change. This is supposed to tell the mapper,
;terminate after the network has stabilized.

-map-once
;-unchanged-maps 2

;-expected-nodes and expected-hosts modify the map-once and
;-unchanged-maps options so that they only take effect when the mapper
;has seen the specified number of nodes or hosts.

;-expected-nodes 466
;-expected-hosts 4


;-unit is the board number the mapper runs on. See the documentation
;for GM and gm_open etc. Most of the time people have only one myrinet
;board in a machine, and its unit number is zero.

-unit 0



;-make-hosts gives the mapper permission to assign GM IDs to
;nodes. Only one mapper in a network should have permission to do this,
;and it should enable -host-file also. Additional mappers should not be
;allowed to assign GM IDs. These extra mappers (passive mappers or
;backup mappers) use existing assigments (either learned from the
;network itself, or taken from a hosts file).

-make-hosts



;there can be more than one mapper running on a myrinet, but only one
;active mapper. An active mapper maps the network and configures it. A
;passive mapper wakes up periodically and tries to become the active
;mapper. A mapper becomes active when its level (this option) is
;highest. If all levels are equal, then the mapper the largest Myrinet
;board ID is the active mapper. Users can use -level to establish a
;ranking of what mappers become active first. Multiple mappers are a
;fault tolerance thing. If a network is split into multiple networks
;(perhaps because of an earthquake, or some a.e.) each new network will end up
;with its own active mapper. When the opposite occurs -- two networks
;join together, GM will become confused if a single GM ID has been
;assigned to more than one node. This is prevented with the -make-hosts
;and -hosts-file option.

-level 0



;-verify tells the mapper to go into "verify" mode after mapping and
;configuring the network. This mode avoids switch compare messages,
;but little else, and its only effect, as far as I can tell, is to
;-make the mapper less responsive to changes in network topology. It
;-should hardly be used.

;-verify



;-remember-nodes tells the mapper to remember what the network looks
;like while it re-maps the network, to improve the mapping speed by
;ordering scouting messages somewhat. It was Glenn's idea, and I think
;it works.

-remember-nodes

;-add-missing-hosts tells the mapper to add missing hosts to switch
; ports currently seen by the mapper as disconnected. The mapper
; remembers the hosts that were attached to these ports previously.
; This option is meant to lessen the impact of hosts falling off the
; network because of dropped host scouting or reply messages during
; congestion.

;-add-missing-hosts

;-configure-missing-hosts works with add-missing-hosts, above. The
;option tells the mapper to send configure messages to, and expect acks
;back from, missing hosts. If the hosts are really missing (and not
;just fallen off the mapper's current view of the network) this option
;will result in the mapper complaining that not all hosts were
;configured, and in fact, the mapper will keep on trying to
;re-configuring after each mapping phase.

;-configure-missing-hosts


; only reconfigure network after a certain number of changed maps have
; been seen.

;-changed-maps 5

;-sore-loser tells the mapper to terminate itself whenever it
;discovers another mapper with a higher priority, as determined by
;the combination of -level and Myrinet board id, as explained by the
;description of the -level option (above).

;-sore-loser



;here are a bunch of options that specify retries and timeouts for
;various kinds of mapping messages. It is beyond the scope of this
;comment to explain what each message type means. But many mapping
;problems can be solved by adjusting these values. Either making them
;about twice or four times larger or smaller. If a network is being
;constantly reconfigured, it may mean the mapper isn't waiting long
;enough for hosts to reply to its scout messages, and so hosts are
;appearing and vanishing from the mapper's view, forcing a
;reconfigure. In this case -scout-time should be doubled. One hint is
;the counters file (see -counters-file above). If ever the "bad
;phases" counter is going up, timeouts should be increased. Timeouts
;are in microseconds

-scout-tries 13
-scout-time 5000
-probe-tries 3
-probe-time 1000
-configure-tries 10
-configure-time 100000

-verify-scout-tries 10
-verify-scout-time 1000
-verify-probe-tries 10
-verify-probe-time 200

;percents to try to increase or decrease timeouts

;-timeout-decrease 10
;-timeout-increase 40

;set timeouts based on measured latencies
;-measure-latency
;-measure-configure-latency

;-active-time tells the mapper how long (in milliseconds) to pause
;before remapping the network.

;-active-time 30000


;-passive-time tells the mapper when it is in passive mode how long
;(in milliseconds) to pause before trying to become the active mapper.

;-passive-time 90000



;-send-many-at-once tells the mapper to send scout and probe messages
;in groups, instead of one at a time. This results in much lower
;time to map a network, but it may be unsafe because of deadlocks or
;it may be unreliable because of congestion.

-send-many-at-once




;-send-many-configures tells the mapper to send configure messages as
;fast as possible, without waiting for acknowledgements for each one.

-send-many-configures




;-interleave-configures sends configure messages evenly. Each host
;might require a number of configure messages. When this option is
;enabled, configure messages are sent out so that each host is sent its
;first message, then each host is sent its second, and so on. This
;will leave a greater number of hosts in a half-configured state for
;longer, but with this option disabled, and the send-many-configures
;option enabled, the mapper can send messages though a path to a single
;host faster than the host can deal with them, which results in very
;visible network blocking -- the configure messages slow down to a crawl.

-interleave-configures

;-resend-configures-in-order resends unacked configure messages such
;that if message N is resent, then messages 0...N-1 have already been
;acked. Some MCPs may work better this way.

;-resend-configures-in-order

;-never-configure-others tells the mapper to only configure itself.
;this is useful when the file_mapper is used to configure a single
;node.
-never-configure-others


;-send-many-compares is like -send-many-configures and
;-send-many-at-once, applied  to switch compare messages. Some people
;-think tht switch compare messages are bit unsafe, and sending many
;-of them at once even more unsafe.

-send-many-compares



;-compare-switches tells the mapper to use switch compare messages to
;distinguish between hostless switches. Without switch compare messages
;the mapper will create ghost switches which must be later resolved
;into single switches using knowlege about the network topology. This
;may result in much longer -- or interminable -- mapping times.

-compare-switches




;-never-configure prevents the mapper from sending configure messages.

;-never-configure



;-max-configure-routes limits the size of each obsolete configure message.

;-max-configure-routes 48

;-max-configure-byte-stream limits the size of each configure message
;payload.

;-max-configure-byte-stream 300



;-find-loops tells the mapper to look for loopback connections on the
;-network. This is an expensive procedure, and most people don't use
;-loopback cables, so it is made an option.

;-find-loops



;-just-set-hostname tells the mapper to just call gm_set_hostname and
;terminate. I'm pretty sure that this is a useless option now.

;-just-set-hostname

;-never-set-hostname tells the mapper not to call gm_set_hostname 

;-never-set-hostname


;-never-end -tells the mapper to never terminate.
;-never-end


;-max-depth tells the mapper never to search beyond a certain depth in mapping
;the network. This is a good option to use if you have
;-compare-switches disabled.

;-max-depth 5



;-warnings tells the mapper to print out important warnings
;progress. On Microsoft Windows NT (tm) the mapper service prints to a
;file called /WINNT/GM/mapper.log

-warnings

;-verbose tells the mapper to print statements describing its
;progress. On Microsoft Windows NT (tm) the mapper service prints to a
;file called /WINNT/GM/mapper.log

;-verbose


;-very-verbose tells the mapper to print out low level information
;about the messages it is sending and waiting for. It is too detailed
;to be of any use except in debugging emergencies, and even then it
;is not really useful.

;-very-verbose



;-thrash tells the mapper to keep remapping and reconfiguring even if
;it isn't necessary. This is for debugging.

;-thrash


;-verify-hosts probably does something stupid or useless. Don't use it.
;-verify-hosts



;-short-host-names tells the mapper to remove the network part of
;hostnames when it is collecting hostnames.

;-short-host-names

;-test-deadlock enables deadlock detecting in the configure
;part of the mapper. If the routes can deadlock, the mapper
;will not configure the network. This option makes sense only if you
;are using a custom route calculator, or the standard calculator with
;-shortest-path enabled.

;-test-deadlock


;increase configure timeouts for every retry
-grow-configure-timeouts

;number of receive buffers. This should be at least 32 not more than GM_NUM_RECV_TOKENS
;-receive-buffers 100

;clear the route table on mapper node before setting new routes
-clear-routes


;now here follows arguments for the simple route calculator.

-route-args

;-save-routes keeps whole route table in memory. It should be used
;when the mapper -interleave-configures option is enabled, otherwise
;routes will be overly recomputed 

-save-routes

;use this if you suspect that the standard, up*/down* (deadlock free)
;routing is giving you bad performance. If you use this option, try
;using -test-deadlock (above).

;-shortest-path

#!/bin/sh
#
# Load static myrinet routes into just myself.
# Assumes two myri cards present on node:  unit 0 and unit 1.
# Requires presence of files static.{hosts,map,routes} as generated
# by a single node with "./mapper static.args".
#
# Add "-verbose" to the -mapper-args to watch it more carefully.
#
# <[email protected]> 9 Feb 2001
#
srcdir=/usr/local/gm/sbin

cd $srcdir
for unit in 0 1 ; do
    echo -n "Myrinet unit $unit "
    ./file_mapper \
	-mapper-args \
	    -map-file static.map \
	    -host-file static.hosts \
	    -never-configure-others \
	    -map-once \
	    -unit $unit \
	-route-args \
	    -routes-file static.routes
done
exit 0