Re: gm_board_info only shows the node itself

Thank you Gary,

Yes, I aware this... and actually, if I do 
mpirun -np 1 ./cpi

then it works without problem.
It means, when I am not using Myrinet, it works...

I think I should try latest version, GM 2.0.8 tomorrow...

Thank you and regards,

----------------------------------------------------
ATSUKO MIYASHITA
Linux Support Center, SWSC, Technical Support - IBM Japan
Tel : 03-3808-8993 ( ext. 1712-8993 )  /  Fax : 03-3664-4893
e-mail : ATSUKOM＠jp.ibm.com
** Moved to Hakozaki - mind that tel # has changed ! ** 




[email protected]
2004/01/13 23:37

 
        To:     Atsuko Miyashita/Japan/IBM@IBMJP
        cc:     "Dr. Markus Fischer" <[email protected]>, [email protected]
        Subject:        Re: [Myrinet] gm_board_info only shows the node itself

 




> But I couldn't make mpirun work because of the message below.

> >[root@hgt001 basic]# mpirun -np 4 ./cpi
> >Process 2 of 4 on hgt001.linux.coc
> >[3]: alloc failed, not enough memory (Fatal Error)
> >Context: <(gmpi_init) gmpi_dma_alloc: dma send buffers>
> >[1]: alloc failed, not enough memory (Fatal Error)
> >Context: <(gmpi_init) gmpi_dma_alloc: dma send buffers>

> I confirmed the memory is not full...

This resembles what I saw when I first started using mpich v2.5.10
and didn't realize that rather than using ~/.gmpi/conf as a machines
file by default,  mpich now uses

whatever_your_mpi_path/share/machines.ch_gm.LINUX

as the default machines file.

Were you already aware of this?



Gary







Atsuko Miyashita <[email protected]> on 01/13/2004 01:27:15 AM

To:   "Dr. Markus Fischer" <[email protected]>
cc:   [email protected] (bcc: Gary Hannon/CSP)
Subject:  Re: [Myrinet] gm_board_info only shows the node itself




Hi Markus,

Thank you for the reply.
These are the output of "gm start" and "gm stop".

>[root@hgt001 root]# service gm stop
>Stopping gm... /etc/init.d/gm: kill: (13599) - No such process
>done.
>[root@hgt001 root]# service gm start
>Starting gm... active mapper... done.

And on the last lines of dmesg, I saw these.

>GM: WARNING:
drivers/linux/gm/gm_arch.c:1368:gm_arch_lock_user_buffer_page():ker
>nel
>GM: trying to register a page with count 0

Sometimes I also see this message in dmesg too.

>GM: Application closed file descriptor while mappings still alive: port
destruct
> delayed

If I should attach some other message, please let me know.
Thank you very much in advance.

(Additional odd thing)
BTW... while I did gm stop/gm start so many times, I suddenly see that the
mapper
process properly worked.

>Port: Status  PID
>   0:   BUSY 14644  (this process [gm_board_info])
>   1:   BUSY 14558
>Route table for this node follows:
>gmID MAC Address                                 gmName Route
>---- ----------------- --------------------------------
---------------------
>   1 00:60:dd:7f:61:be                           hgt001 (this node)
>   2 00:60:dd:7f:61:fc                           hgt002 84 (mapper)

But I couldn't make mpirun work because of the message below.

>[root@hgt001 basic]# mpirun -np 4 ./cpi
>Process 2 of 4 on hgt001.linux.coc
>[3]: alloc failed, not enough memory (Fatal Error)
>Context: <(gmpi_init) gmpi_dma_alloc: dma send buffers>
>[1]: alloc failed, not enough memory (Fatal Error)
>Context: <(gmpi_init) gmpi_dma_alloc: dma send buffers>

I confirmed the memory is not full...
After I rebooted the system, again I see the mapper process doesn't work
anymore.

>[root@hgt001 basic]# cat /proc/meminfo
>        total:    used:    free:  shared: buffers:  cached:
>Mem:  4219338752 400519168 3818819584        0 63873024 178229248
>Swap: 2089209856        0 2089209856
>MemTotal:      4120448 kB
>MemFree:       3729316 kB
>MemShared:           0 kB
(Cont...)

Do you have any idea why my environment is such unstable ?????


----------------------------------------------------
ATSUKO MIYASHITA
Linux Support Center, SWSC, Technical Support - IBM Japan
Tel : 03-3808-8993 ( ext. 1712-8993 )  /  Fax : 03-3664-4893
e-mail : ATSUKOM
＠jp.ibm.com
** Moved to Hakozaki - mind that tel # has changed ! **




"Dr. Markus Fischer" <[email protected]>
2004/01/11 07:08


To:     Atsuko Miyashita/Japan/IBM@IBMJP
cc:     [email protected]
Subject:        Re: [Myrinet] gm_board_info only shows the node
itself



I guess you should provide the output of

/etc/init.d/gm start

and provide the output of the last lines of dmesg

Markus

Atsuko Miyashita wrote:

>
> Hi,
>
> Thank you for the reply.
> Yes, I run the mapper. I did "gm stop" and "gm start" many times.
>
> Actually, I noticed mapper process hasn't been started - when I
> executed "gm stop", it says there's nothing to kill.
> Attached is my /tmp/gm_log.root, a log of "gm start".
> According to this log, it seems that "gm start" tried to run the
> mapper and writes the pid into /var/run/gm_mapper/pid.0 file, but
> mapper died soon.
>
> I powered off the Myrinet switch and leave it for a while(15min or
> so), then power it on again, but it remains the same.
>
> I am at a loss what to do next... has anyone had similar experience ?
>
> Sat Jan 10 17:28:12 JST 2004
> /etc/init.d/gm start
> root@hpc001> test xgm != x
> Starting gm...
> root@hpc001> /opt/gm/bin/gm_board_info
> GM build ID is "2.0.6_Linux_rc20030908173009PDT
> root@hpc001:/usr/local/gm-2.0.6_Linux Wed Dec 24 16:34:42 JST 2003."
> No boards found
> root@hpc001> rmmod --verbose gm
> Checking gm for persistent data
> rmmod: module gm is not loaded
> root@hpc001> insmod --verbose /lib/modules/2.4.18-3bigmem/gm/gm.o
> Warning: loading /lib/modules/2.4.18-3bigmem/gm/gm.o will taint the
> kernel: non-GPL license - Myricom
> Using /lib/modules/2.4.18-3bigmem/gm/gm.o
> Symbol version prefix 'smp_'
> root@hpc001> /opt/gm/bin/gm_board_info
> GM build ID is "2.0.6_Linux_rc20030908173009PDT
> root@hpc001:/usr/local/gm-2.0.6_Linux Wed Dec 24 16:34:42 JST 2003."
>
>
> Board number 0:
> lanai_clockval = 0x082082a0
> lanai_cpu_version = 0x0900 (LANai9.0)
> lanai_board_id = 00:60:dd:7f:61:be
> lanai_sram_size = 0x00200000 (2048K bytes)
> max_lanai_speed = 134 MHz
> product_code = 109
> serial_number = 89434
> (should be labeled: "M3S-PCI64B-2-89434")
> LANai time is 0x000072c15 ticks, or about 0 minutes since reset.
> Mapper is 00:00:00:00:00:00.
> Map version is 0.
> 0 hosts.
> Network is NOT fully configured.
> This node is "hpc001"
> Board has room for 16 ports, 1600 nodes/routes, 16384 cache entries
> Port token cnt: send=61, recv=254
> Port: Status PID
> 0: BUSY 2742 (this process [gm_board_info])
> Route table for this node follows:
> gmID MAC Address gmName Route
> ---- ----------------- --------------------------------
> ---------------------
> 1 00:60:dd:7f:61:be hpc001 (this node)
> root@hpc001> /opt/gm/bin/gm_set_name --board=0 --host-name=hpc001
> Could not open board 0, port 1 : out of memory
> root@hpc001> /opt/gm/sbin/gm_mapper --unit=0
> --daemon-pid-file=/var/run/gm_mapper/pid.0
> --verbose-file=/var/run/gm_mapper/verbose.0
> --map-file=/var/run/gm_mapper/map.0
> Storing daemon PID in file "/var/run/gm_mapper/pid.0"
> done.
>
> Thank you very much in advance !
>
>
> ----------------------------------------------------
> ATSUKO MIYASHITA
> Linux Support Center, SWSC, Technical Support - IBM Japan
> Tel : 03-3808-8993 ( ext. 1712-8993 ) / Fax : 03-3664-4893
> e-mail : ATSUKOM＠jp.ibm.com
> ** Moved to Hakozaki - mind that tel # has changed ! **
>
>
>
>                "Dr. Markus Fischer" <[email protected]>
>
> 2004/01/10 05:32
>
>
> To: Atsuko Miyashita/Japan/IBM@IBMJP
> cc: [email protected]
> Subject: Re: [Myrinet] gm_board_info only shows the node itself
>
>
>
> Have you run the mapper ?
> (active mappers will shop up using port 1)
>
> /etc/init.d/gm start
>
> will start mappers.
>
> Markus
>
> Atsuko Miyashita wrote:
>
> >
> > Hi,
> >
> > I've got a problem that GM mapper only sees the node itself, and
> > doesn't communicate each other.
> > I appreciate if anybody has a clue/suggestion to this... Thank you in
> > advance !
> >
> > I am currently using GM 2.0.6 and Myrinet "B" card with serial cable.
> > I have two nodes(hpc001/hpc002) with Linux 2.4.18-3bigmem(RedHat7.3)
> > installed.
> > I know the combination of driver/card is not the good one, but this is
> > a test environment for some software.
> >
> > I noticed that I couldn't use MPICH-GM properly, and this was probably
> > because of driver layer.
> > gm_board_info only shows the node itself, and doesn't show the another
> > node.
> > I did this several times, but the situation remains the same.
> > service gm stop
> > service gm start
> >
> > Also, I changed the physical port(of the switch), but it didn't solve
> > the problem.
> >
> > The odd thing is that I could use the environment without any problem
> > for several days.
> > I tested the MPICH-GM with its sample program, cpi, and saw it works
> > without problem.
> > I left the machines for a couple of days and then returned, and saw
> > this has happend.
> >
> > Does anyone have any idea or suggestion, at least how to diag this ?
> >
> > These are the output of gm_board_info on the each nodes.
> >
> > [root@hpc001 root]# gm_board_info
> > GM build ID is "2.0.6_Linux_rc20030908173009PDT
> > root@hpc001:/usr/local/gm-2.0.6_
> > Linux Wed Dec 24 16:34:42 JST 2003."
> > Board number 0:
> > lanai_clockval = 0x082082a0
> > lanai_cpu_version = 0x0900 (LANai9.0)
> > lanai_board_id = 00:60:dd:7f:61:be
> > lanai_sram_size = 0x00200000 (2048K bytes)
> > max_lanai_speed = 134 MHz
> > product_code = 109
> > serial_number = 89434
> > (should be labeled: "M3S-PCI64B-2-89434")
> > LANai time is 0x058a585f0 ticks, or about 11 minutes since reset.
> > Mapper is 00:00:00:00:00:00.
> > Map version is 0.
> > 0 hosts.
> > Network is NOT fully configured.
> > This node is "hpc001"
> > Board has room for 16 ports, 1600 nodes/routes, 16384 cache entries
> > Port token cnt: send=61, recv=254
> > Port: Status PID
> > 0: BUSY 1503 (this process [gm_board_info])
> > Route table for this node follows:
> > gmID MAC Address gmName Route
> > ---- ----------------- --------------------------------
> > ---------------------
> > 1 00:60:dd:7f:61:be hpc001 (this node)
> >
> > [root@hpc002 root]# gm_board_info
> > GM build ID is "2.0.6_Linux_rc20030908173009PDT
> > root@hpc002:/usr/local/gm-2.0.6_
> > Linux Wed Dec 24 17:48:24 JST 2003."
> > Board number 0:
> > lanai_clockval = 0x082082a0
> > lanai_cpu_version = 0x0900 (LANai9.0)
> > lanai_board_id = 00:60:dd:7f:61:fc
> > lanai_sram_size = 0x00200000 (2048K bytes)
> > max_lanai_speed = 134 MHz
> > product_code = 109
> > serial_number = 89372
> > (should be labeled: "M3S-PCI64B-2-89372")
> > LANai time is 0x08e6f211a ticks, or about 17 minutes since reset.
> > Mapper is 00:00:00:00:00:00.
> > Map version is 0.
> > 0 hosts.
> > Network is NOT fully configured.
> > This node is "hpc002"
> > Board has room for 16 ports, 1600 nodes/routes, 16384 cache entries
> > Port token cnt: send=61, recv=254
> > Port: Status PID
> > 0: BUSY 1692 (this process [gm_board_info])
> > Route table for this node follows:
> > gmID MAC Address gmName Route
> > ---- ----------------- --------------------------------
> > ---------------------
> > 1 00:60:dd:7f:61:fc hpc002 (this node)
> >
> > Thank you in advance for your help !!!
> >
> > ----------------------------------------------------
> > ATSUKO MIYASHITA
> > Linux Support Center, SWSC, Technical Support - IBM Japan
> > Tel : 03-3808-8993 ( ext. 1712-8993 ) / Fax : 03-3664-4893
> > e-mail : ATSUKOM＠jp.ibm.com
> > ** Moved to Hakozaki - mind that tel # has changed ! **
> >
>
>------------------------------------------------------------------------
> >
> >_______________________________________________
> >Myrinet mailing list
> >[email protected]
> >http://email.osc.edu/mailman/listinfo/myrinet
> >
> >
>
>
>
>









_______________________________________________
Myrinet mailing list
[email protected]
http://email.osc.edu/mailman/listinfo/myrinet





#### att1.htm has been removed from this note on January 14 2004 by Atsuko 
Miyashita

_______________________________________________
Myrinet mailing list
[email protected]
http://email.osc.edu/mailman/listinfo/myrinet