Re: [myrinet] AS500 with Myrinet card [was: Low bandwidth]

From gm_debug, you are using a 64bit / 33MHz PCI bus. Can you find
a 64bit / 66MHz PCI slot? This will give you higher RW bandwidth.

Karen Wang
[email protected]


On Wed, 29 Aug 2001, Francesco Lovergine wrote:

> Ok, we moved cards on the top slot (long slot 66Mhz).
> Performances are better but well far from theoretical ones.
>
> This is for gm_debug:
>
> DMA rate for 8192 Byte transfers (64bit / 33MHz bus)
> 1st: 8 pages from bogus sdma pg, 8 to bogus rdma
>         bus_read  (send) = 174 MBytes/s
>         bus_write (recv) = 92 MBytes/s
>
> DMA rate for 8192 Byte transfers (64bit / 33MHz bus)
> 2nd: sdma a page, rdma a page 8 times
>         bus_read  (send) = 174 MBytes/s
>         bus_write (recv) = 91 MBytes/s
>
> DMA rate for 8192 Byte transfers (64bit / 33MHz bus)
> 3rd: sdma and rdma to/from alternating pages (coarse grain)
>         bus_read  (send) = 186 MBytes/s
>         bus_write (recv) = 91 MBytes/s
>
> DMA rate for 8192 Byte transfers (64bit / 33MHz bus)
> 4th: sdma and rdma to/from alternating pages (fine grain)
>         bus_read  (send) = 195 MBytes/s
>         bus_write (recv) = 92 MBytes/s
>
>
> And gm_allsize gives ~84MB/sec of max throughput.
> Better than before, but quite low. This is for
> Compaq/Digital AlphaStation 500 with Linux kernel 2.2.19.
> And apparently there is not any margin for improving.
>
> This asymmetric transfers are extremely amazing...
> Could this depend on OS, also?
> What about using Tru64 instead of Linux?
>
>
> On Thu, Aug 02, 2001 at 09:59:01AM +0200, Francesco Lovergine wrote:
> >
> > Thank you, we will do this asap. When done I`ll send an update to this list.
> >
> > On Wed, Aug 01, 2001 at 12:40:55PM -0400, Tenhave, Tim wrote:
> > > Hi,
> > >
> > > I would try to move the Myrinet card to slot 7 - the top slot (hopefully 64
> > > bit in your system) and see if anything improves.  I do know the bottom 2
> > > slots will possibly suffer issues with interrupts and overall throughput
> > > since they "share" resources.
> > >
> > > I also did some digging into what is left of the old Digital internal
> > > "notes" areas and confirmed the PCI performance best seen was 111MB/sec DMA
> > > read, and 65MB/sec DMA write.  This was in a 32 bit slot.  I did not find
> > > any data for the 64 bit slot that appeared as the top slot in the later
> > > AS500-400 and -500 machines.
> > >
> > > Tim
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: Francesco Lovergine [mailto:[email protected]]
> > > Sent: Wednesday, August 01, 2001 3:56 AM
> > > To: Tenhave, Tim
> > > Cc: Francesco Lovergine; [email protected]
> > > Subject: Re: [myrinet] RE: Re: Low bandwidth
> > >
> > >
> > > On Tue, Jul 31, 2001 at 12:51:14PM -0400, Tenhave, Tim wrote:
> > > > Hi,
> > > >
> > > > I would try moving the Myrinet board into other slots.  I do not have
> > > access
> > > > to an AS500 today, but I do know it contained 2 (or 3) PCI buses and you
> > > may
> > > > be competing for resources from other devices in the bus you are on.  You
> > > > can help yourself determine the nature of your bus by using the SRM "show
> > > > config | more" command and attempt to move your Myrinet board to a quieter
> > > > bus.  Realize the slot numbers provided are logical slots and not physical
> > > > slots.
> > > >
> > > > If you need more details, send email.
> > > >
> > > > Tim
> > > >
> > > > Compaq/DEC Alpha guy who uses Myrinet too.
> > > >
> > >
> > > this is the show config output:
> > >
> > >
> > >
> > > PCI Bus
> > >      Bus 00  Slot 06: DECchip 21040 Network Controller
> > >                                    ewa0.0.0.6.0
> > > 00-00-F8-22-F9-E2
> > >
> > >      Bus 00  Slot 07: Cateyes - 4D40T
> > >      Bus 00  Slot 08: DE500-BA Network Controller
> > >                                    ewb0.0.0.8.0
> > > 00-00-F8-10-C9-63
> > >
> > >      Bus 00  Slot 09: ISP1020 Scsi Controller
> > >                                    pka0.7.0.9.0          SCSI Bus ID 7
> > >                                    dka0.0.0.9.0           RZ29B
> > >                                    dka400.4.0.9.0         RRD45
> > >
> > >      Bus 00  Slot 10: Intel   8275EB PCI to Eisa Bridge
> > >
> > >
> > >      Bus 00  Slot 11: Vendor: 14c1  Device: 8043
> > >      Bus 00  Slot 12: Cateyes - VGA Daughter Card
> > >
> > >
> > >
> > > I could remove the powerstorm graphic card (slots 7/12).
> > > Currently only slot8 is used as fastethernet controller.
> > > Any hints?
> > >
> > >
> >
> > --
> > Francesco P. Lovergine
>
> --
> Francesco P. Lovergine
>
>