RE: AS500 with Myrinet card [was: Low bandwidth]

Hi Francesco,

In my research for my earlier replies, I do not believe the operating system
plays any significant role in the results.  It appears to be the PCI design
used.  

You should know the newer EV6 based Alphas do not use this design and are a
bit more even in their results.

Tim


-----Original Message-----
From: Francesco Lovergine [mailto:[email protected]]
Sent: Wednesday, August 29, 2001 10:05 AM
To: Tenhave, Tim; [email protected]
Subject: AS500 with Myrinet card [was: Low bandwidth]


Ok, we moved cards on the top slot (long slot 66Mhz).
Performances are better but well far from theoretical ones.

This is for gm_debug:

DMA rate for 8192 Byte transfers (64bit / 33MHz bus)
1st: 8 pages from bogus sdma pg, 8 to bogus rdma
        bus_read  (send) = 174 MBytes/s
        bus_write (recv) = 92 MBytes/s

DMA rate for 8192 Byte transfers (64bit / 33MHz bus)
2nd: sdma a page, rdma a page 8 times
        bus_read  (send) = 174 MBytes/s
        bus_write (recv) = 91 MBytes/s

DMA rate for 8192 Byte transfers (64bit / 33MHz bus)
3rd: sdma and rdma to/from alternating pages (coarse grain)
        bus_read  (send) = 186 MBytes/s
        bus_write (recv) = 91 MBytes/s

DMA rate for 8192 Byte transfers (64bit / 33MHz bus)
4th: sdma and rdma to/from alternating pages (fine grain)
        bus_read  (send) = 195 MBytes/s
        bus_write (recv) = 92 MBytes/s


And gm_allsize gives ~84MB/sec of max throughput.
Better than before, but quite low. This is for
Compaq/Digital AlphaStation 500 with Linux kernel 2.2.19.
And apparently there is not any margin for improving.

This asymmetric transfers are extremely amazing...
Could this depend on OS, also? 
What about using Tru64 instead of Linux? 


On Thu, Aug 02, 2001 at 09:59:01AM +0200, Francesco Lovergine wrote:
> 
> Thank you, we will do this asap. When done I`ll send an update to this
list.
> 
> On Wed, Aug 01, 2001 at 12:40:55PM -0400, Tenhave, Tim wrote:
> > Hi,
> > 
> > I would try to move the Myrinet card to slot 7 - the top slot (hopefully
64
> > bit in your system) and see if anything improves.  I do know the bottom
2
> > slots will possibly suffer issues with interrupts and overall throughput
> > since they "share" resources.  
> > 
> > I also did some digging into what is left of the old Digital internal
> > "notes" areas and confirmed the PCI performance best seen was 111MB/sec
DMA
> > read, and 65MB/sec DMA write.  This was in a 32 bit slot.  I did not
find
> > any data for the 64 bit slot that appeared as the top slot in the later
> > AS500-400 and -500 machines.
> > 
> > Tim  
> > 
> > 
> > 
> > -----Original Message-----
> > From: Francesco Lovergine [mailto:[email protected]]
> > Sent: Wednesday, August 01, 2001 3:56 AM
> > To: Tenhave, Tim
> > Cc: Francesco Lovergine; [email protected]
> > Subject: Re: [myrinet] RE: Re: Low bandwidth
> > 
> > 
> > On Tue, Jul 31, 2001 at 12:51:14PM -0400, Tenhave, Tim wrote:
> > > Hi,
> > > 
> > > I would try moving the Myrinet board into other slots.  I do not have
> > access
> > > to an AS500 today, but I do know it contained 2 (or 3) PCI buses and
you
> > may
> > > be competing for resources from other devices in the bus you are on.
You
> > > can help yourself determine the nature of your bus by using the SRM
"show
> > > config | more" command and attempt to move your Myrinet board to a
quieter
> > > bus.  Realize the slot numbers provided are logical slots and not
physical
> > > slots.
> > > 
> > > If you need more details, send email.
> > > 
> > > Tim
> > > 
> > > Compaq/DEC Alpha guy who uses Myrinet too.
> > >   
> > 
> > this is the show config output:
> > 
> > 
> > 
> > PCI Bus
> >      Bus 00  Slot 06: DECchip 21040 Network Controller
> >                                    ewa0.0.0.6.0
> > 00-00-F8-22-F9-E2
> >  
> >      Bus 00  Slot 07: Cateyes - 4D40T
> >      Bus 00  Slot 08: DE500-BA Network Controller
> >                                    ewb0.0.0.8.0
> > 00-00-F8-10-C9-63
> >  
> >      Bus 00  Slot 09: ISP1020 Scsi Controller
> >                                    pka0.7.0.9.0          SCSI Bus ID 7
> >                                    dka0.0.0.9.0           RZ29B
> >                                    dka400.4.0.9.0         RRD45
> >  
> >      Bus 00  Slot 10: Intel   8275EB PCI to Eisa Bridge
> >  
> >  
> >      Bus 00  Slot 11: Vendor: 14c1  Device: 8043
> >      Bus 00  Slot 12: Cateyes - VGA Daughter Card
> > 
> > 
> > 
> > I could remove the powerstorm graphic card (slots 7/12).
> > Currently only slot8 is used as fastethernet controller.
> > Any hints?
> > 
> > 
> 
> -- 
> Francesco P. Lovergine

-- 
Francesco P. Lovergine