RE: mass storage recommendations

> What's the exact configuration of the 4T RAID? Are you using
> mdadm/raidtools? In what RAID level?
> When reading the description you gave, I would guess you're running
some
> kind of RAID5 over 500GB disks, with a total of 8 disks, creating
> 4000-500=3500MB of effective storage.

You're exactly right in all counts.

> 100T sounds like a lot of space. Assuming you're using 500G disks, you
> need 200 (!!) disks in RAID.

Yep. And we're eyeing the latest 750G disks, that would only take 133
disks, with 8 per box we're talking 17 machines to host 100T

> > So we see these two factors opposing each other: (1) large
> > easy-to-manage partition and (2) failover / fault tolerance.
> Remembering the things I wrote before, I would suggest something like
> this:
> 
> disk1 \
> disk2 |  RAID5
> disk3 |  1.5T
> disk4 /
> 
> disk1 \
> disk2 |  RAID5
> disk3 |  1.5T
> disk4 /
> 
> disk1 \
> disk2 |  RAID5
> disk3 |  1.5T
> disk4 /
> 

I was thinking that, only with the 8 (or maybe 10) disks per physical
box, then tying them together with one box in the front, and it doing
raid5 on the enbd disks it pulls in. In this way it would actually take
4 disks (two on two different systems) or 2 whole systems to fail before
the data wouldn't be accessible (or the front end box, which that's
simple enough to fix, we just drop another front end in place enbd the
backends into the new frontend).

> etc.
> 
> And combine those sets using a ENBD client (on a frontend machine).
> However, this would require about 67 RAID5 ENBD servers. I'm not
really
> sure, but I don't think this is really possible using ENBD. It just
> isn't designed to fit in such configuration.

With my idea of 8 disks per box and raid5 on raid5 it would take 30
machines to run 100T

> I'm using ENBD for a storage clusters of 3-5 machines, effectively
> creating 500G to 2T per cluster (more is possible, but not needed).
I've
> considered using OpenAFS and other distributed file system solutions,
> but those would cause too much overhead in my situation. However,
> considering your needs, I think you should be looking for something
like
> that.

I'll check out OpenAFS. Any other distributed file systems you can
recommend?

> It's possible that Peter (main developer of ENBD) has a completely
> different opinion on this topic, so you shouldn't immediately draw
your
> conclusions on my thoughts. They're just thoughts...

Cool. Let's see what Peter says too.

Thanks for your input, it's exactly what I am looking for.