RE: mass storage recommendations

On Tue, 2006-05-30 at 21:57 -0500, Richard Pickett wrote:
> > What's the exact configuration of the 4T RAID? Are you using
> > mdadm/raidtools? In what RAID level?
> > When reading the description you gave, I would guess you're running
> some
> > kind of RAID5 over 500GB disks, with a total of 8 disks, creating
> > 4000-500=3500MB of effective storage.
> 
> You're exactly right in all counts.
> 
> > 100T sounds like a lot of space. Assuming you're using 500G disks, you
> > need 200 (!!) disks in RAID.
> 
> Yep. And we're eyeing the latest 750G disks, that would only take 133
> disks, with 8 per box we're talking 17 machines to host 100T

when u have 100TB from 750GB disks, raid5 is not enough for you. u need
to look at raid6.

> 
> > > So we see these two factors opposing each other: (1) large
> > > easy-to-manage partition and (2) failover / fault tolerance.
> > Remembering the things I wrote before, I would suggest something like
> > this:
> > 
> > disk1 \
> > disk2 |  RAID5
> > disk3 |  1.5T
> > disk4 /
> > 
> > disk1 \
> > disk2 |  RAID5
> > disk3 |  1.5T
> > disk4 /
> > 
> > disk1 \
> > disk2 |  RAID5
> > disk3 |  1.5T
> > disk4 /
> > 
> 
> I was thinking that, only with the 8 (or maybe 10) disks per physical
> box, then tying them together with one box in the front, and it doing
> raid5 on the enbd disks it pulls in. In this way it would actually take
> 4 disks (two on two different systems) or 2 whole systems to fail before
> the data wouldn't be accessible (or the front end box, which that's
> simple enough to fix, we just drop another front end in place enbd the
> backends into the new frontend).

raid5 over raid5 will be quite slow.

will u front end become a performance bottleneck? what kind of io
workload u are expecting on this front end?


> 
> > etc.
> > 
> > And combine those sets using a ENBD client (on a frontend machine).
> > However, this would require about 67 RAID5 ENBD servers. I'm not
> really
> > sure, but I don't think this is really possible using ENBD. It just
> > isn't designed to fit in such configuration.
> 
> With my idea of 8 disks per box and raid5 on raid5 it would take 30
> machines to run 100T
> 
> > I'm using ENBD for a storage clusters of 3-5 machines, effectively
> > creating 500G to 2T per cluster (more is possible, but not needed).
> I've
> > considered using OpenAFS and other distributed file system solutions,
> > but those would cause too much overhead in my situation. However,
> > considering your needs, I think you should be looking for something
> like
> > that.
> 
> I'll check out OpenAFS. Any other distributed file systems you can
> recommend?
> 
> > It's possible that Peter (main developer of ENBD) has a completely
> > different opinion on this topic, so you shouldn't immediately draw
> your
> > conclusions on my thoughts. They're just thoughts...
> 
> Cool. Let's see what Peter says too.
> 
> Thanks for your input, it's exactly what I am looking for.
> 
> _______________________________________________
> ENBD mailing list
> [email protected]
> http://lists.community.tummy.com/mailman/listinfo/enbd