Re: [myrinet] Re: Re: Low bandwidth

Hi again,
at
http://citeseer.nj.nec.com/1690.html
there is a paper which investigates some issues concerning PCI bandwidth.
quote:

DMA read
This kind of transaction resmebles DMA write except that, as it involves a
roundtrip to the memory system before first data can be returned, the
latency to initial data can be very large. On high-end servers like the
AlphaServer 8400, where the memory is in a different subsystem to the PCI
host bridge, the latency is many tens of PCI cycles, even though the
bandwidth of the memory backplane is 2 GB/s.
...
....
Platforms systematically disconnecting transactions like the AlphaStation
200 or 400 can reach only 67 MB/s, while standart long bursting platforms
reach 120 MB/s to more than 200MB/s for 64-bit PCI machines.

I don't know whether your AlphaStation500 is different thatn the 400 or
200, but if not, you're probably not going to get any better...

cheers,
Clemens


On Wed, 1 Aug 2001, Francesco Lovergine wrote:

> On Tue, Jul 31, 2001 at 09:31:10AM -0400, Pete Wyckoff wrote:
> > [email protected] said:
> > > I just built up my first beowulf cluster with the latest PCI64B-2
> > > cards (PCI64C was available just a month later our order :-().
> > > Hosts are all DEC Alphastation 500/500 (alcor arch). OS is Linux
> > > with kernel 2.2.19. Buses are 32bits @ 33Mhz so my guess is at least
> > > a 100-130MBs of throughput. In fact, reality is well different...
> > > gm_allsize gives me a maximum bandwith of 56MBs.
> >
> > What does gm_debug say for a DMA rate?
>
>
> That's interesting:
>
> Opening board 0, port 0
>
> DMA rate for 8192 Byte transfers (32bit / 33MHz bus)
> 1st: 8 pages from bogus sdma pg, 8 to bogus rdma
>         bus_read  (send) = 126 MBytes/s
>         bus_write (recv) = 60 MBytes/s
>
> DMA rate for 8192 Byte transfers (32bit / 33MHz bus)
> 2nd: sdma a page, rdma a page 8 times
>         bus_read  (send) = 126 MBytes/s
>         bus_write (recv) = 60 MBytes/s
>
> DMA rate for 8192 Byte transfers (32bit / 33MHz bus)
> 3rd: sdma and rdma to/from alternating pages (coarse grain)
>         bus_read  (send) = 126 MBytes/s
>         bus_write (recv) = 60 MBytes/s
>
> DMA rate for 8192 Byte transfers (32bit / 33MHz bus)
> 4th: sdma and rdma to/from alternating pages (fine grain)
>         bus_read  (send) = 126 MBytes/s
>         bus_write (recv) = 60 MBytes/s
>
> First two words of LANai globals: 0x10000000, 0xcafebabe
> Some counters from LANai
>
> netsend_cnt      1810209
> netrecv_cnt      1439851
> error counters
>   bad_header_cnt                                      0
>   bad_length_cnt                                      0
>   bad_type_cnt                                        0
>   badcrc_cnt                                          0
>   badroute_cnt                                        0
>   bogus_header_cnt                                    0
>   drop_cnt                                            5
>   handle_connection_reset_request_cnt                 2
>   misrouted_cnt                                       0
>   nack_cnt                                            0
>   nack_down_cnt                                       0
>   nack_ignore_close_connection_cnt                    0
>   nack_ignore_open_connection_cnt                     1
>   nack_ignored_cnt                                   17
>   nack_normal_cnt                                    17
>   nack_receive_close_connection_cnt                   0
>   nack_receive_open_connection_cnt                    3
>   nack_received_cnt                                  41
>   nack_reject_cnt                                     0
>   nack_send_close_connection_cnt                      0
>   nack_send_nothing1_cnt                              0
>   nack_send_nothing2_cnt                              0
>   nack_send_open_connection_cnt                       2
>   no_match_for_datagram_recv_cnt                      0
>   no_match_for_ether_recv_cnt                         0
>   no_match_for_reliable_recv_cnt                      0
>   no_match_for_raw_recv_cnt                           0
>   out_of_sequence_cnt                                 2
>   resend_cnt                                         19
>   short_mapper_config_packet_cnt                      0
>   short_mapper_packet_cnt                             0
>   short_mapper_scout_packet_cnt                       0
>   short_packet_cnt                                    0
>   used_bogus_send_cnt                                 0
>   used_bogus_recv_cnt                                 0
>   zero_len_cnt                                        2
>
>        packet_sexno 0000 0001
>        ack_sexno 03e9 03e8
>
>
> port counters
>   port[2].active_subport_cnt                          0
>
>
> So there is a difference in rcv/snd throughput !?
>
>

-- 
________________________________________________________________
Clemens Adler                      phone: (631)344-8334
Bldg. 510A                         email: [email protected]
Brookhaven National Lab
Upton N.Y. 11973                   Office: Bldg. 118  Rm. 103
________________________________________________________________