Re: OpenSSI & Infiniband

> -----Original Message-----
> From: Vincent Diepeveen [mailto:[email protected]] 
> Sent: Friday, June 16, 2006 2:02 AM
> To: Barry, Christopher; [email protected]
> Subject: Re: [SSI-users] OpenSSI & Infiniband
> 
> 
> ----- Original Message ----- 
> 
> > Unfortunately for Quadrics and Myrinet both, their days 
> look numbered,
> > and their applications more and more fringe. It's fairly 
> obvious they
> > are on the decline now that IB is here.
> >
> > I can probably help with getting OSSI up again w/ IB. I've 
> been wanting
> > to do this for several years, and I think I'll be having 
> some time to
> > spend coming up. Let me know how I can help, what others 
> have done thus
> > far, etc. Get me up to speed on it's current status.
> > Thanks,
> > Chris Barry
> 
> I'm not so sure of that.
> 
> Both quadrics and infiniband are pretty expensive solutions 
> for highend 
> networks.
> Myri on other hand is real cheap and dominating market right now.
> 
> Quadrics is probably together with infiniband scaling very 
> well to large 
> networks,
> yet quadrics has a very fast one way pingpong latency.
> 
> On paper Myri has a very good one too, yet of course scales 
> not so well when 
> all
> nodes are broadcasting small messages at the same time.
> 
> Infinibands one way pingpong latency really is ugly for a 
> highend network 
> card.
> 
> You just can't use that network for short message software 
> that needs quick 
> latency.
> 
> So quadrics has the advantages of both, but also a higher 
> price than both 
> other.
> Price of quadrics is real ugly at card level.
> 
> Where the network (which is most expensive at big networks) 
> is similar 
> priced,
> 1 card is soon $1000 a piece at quadrics versus $500 for myri.
> 
> Additional it has a shared memory library that's real nice 
> using the shmem 
> library.
> 
> Yet if you look to price per node it's not that much of a huge price 
> compared to the cost of a node itself.
> 
> A highend cluster that you'll see with a default outdated P4 
> dual Xeon 3 Ghz 
> "prescott core" that uni's order
> soon you're thinking in 5000-6000 dollar a node and adding 
> another 1500 
> dollar a node for
> a network is not that much of a big pain in that case.
> 
> On other hand look who's talking, i'm trying to build some 
> cheapo cluster 
> now.
> 
> For highend networks bigger than 2 nodes always the problem 
> is: "how to get 
> a cheap switch".
> 
> So in the end myri outsells them all as it's cheaper a node simply.
> Most scientists wouldn't be able to tell their grandmother 
> what network is 
> faster than the other.
> Basically things are so technical that every manufacturer has 
> a story of his 
> own and in the end
> what happens is that the company building the cluster will 
> put in of course 
> the cheapest highend
> network, as they can earn more a node in that case.
> 
> That's how real world life works simplistically said.
> 
> Bit sad, at the older cards infiniband has 128MB cache, 
> quadrics has 64MB 
> cache, Myri has most cards like 4MB cache.
> 
> Quite a gap.
> 
> Yes there is 8MB versions but those were $1500 a card or so 
> and probably 
> some newer card by now released with even more for > $1k.
> 
> What will happen of course real soon is that windows will 
> take over which 
> sucks. The highend company who manages to get supported by
> windows in kernel of windows cluster, is going to sell major 
> league and will 
> by means of a low price per port obviously doom all other
> manufacturers real soon.
> 
> Vincent 
> 
>

What IB has, and the others do not (not to mention bandwidth) is the
beauty of IO virtualization.

With a single 12x DDR (Double Data Rate) connection in a node (that's
60Gb/s!!!), you also get any number of virtual Ethernet interfaces AND
any number of virtual fibrechannel interfaces - all happening isolated
in the same pipe. That's a huge thing when you think about it. The
actual GbE and 10Gbe interfaces, and 2Gb and 4Gb FC interfaces live in
the switch proper. Every single node can mount the same slice from a
storage device. And with the emergence of direct IB storage, the problem
is getting enough spindles in the storage facility to not starve the
pipe!

IB is definitely going to kick everyone's ass for not just huge, low
latency clusters, but in the Enterprise, enabling a 'single wire'
infrastructure that enables extremely complex IB/Eth/Fc networks to be
simplified and accelerated. We created with Oracle a new protocol called
RDS, (Reliable Datagram Service) that is the communication method of
choice for Oracle RAC, their new database clustering technology, using
commodity boxes tied together with IB, and it smokes.

I've worked at SilverStorm (formerly InfiniCon Systems) since it's
inception - over 5 years now. I've seen IB be totally misunderstood, and
it almost disappeared - almost everyone in the space folded years ago.
It's ripe now, people are understanding it better now, it's more mature
and it works.

In my opinion, IB is THE interconnect that is ideal for OSSI, it's fast,
does RDMA extremely well, has built-in sharedIO functionality. It's
going to make OSSI a major player in the Enterprise. I'm convinced of
it.


-C