Re: mass storage recommendations

Hi Richard,


> Currently we have a web interface that serves up files from a number of
> raid boxes behind it (`mount`ed on the web server). Currently the raids
> are 4T (3.5T usable). It’s a simple enough solution, if a raid box
> starts acting flaky we pull it out and can drop a replacement in, copy
> the data over and it’s back up and running. The down side is the
> management. Each box can host a limited number of clients, and if one
> client starts growing beyond the size of that box we’re in trouble (or
> at least in for a management headache). It would be easier to have one
> huge raid, say 100T, and then clients that grow wouldn’t really be a
> management problem.
What's the exact configuration of the 4T RAID? Are you using
mdadm/raidtools? In what RAID level?
When reading the description you gave, I would guess you're running some
kind of RAID5 over 500GB disks, with a total of 8 disks, creating
4000-500=3500MB of effective storage.


> With that in mind we were exploring the idea of taking a layered
> approach to the raids and their devices. We could easily have a “front”
> box that enbd-pulls in a large number of devices that are running on a
> number of “back” boxes, then raid across those devices.
100T sounds like a lot of space. Assuming you're using 500G disks, you
need 200 (!!) disks in RAID.


> Another step after this would be to make an LVM that runs on top of
> several of these raided systems, so if we need to grow the usable space
> we just add more hardware to the LVM.
Sounds possible.


> It is easier to manage because all of the hosted files would exist on
> one huge mount point. The problem is the fault-tolerance and recovery.
> With our current solution we can replace an existing box, files and all,
> within a few hours. With the LVM-on-RAID-on-ENBD if two units (whether
> that’s two HD if the “front” box enbd-pulls each device in, or two
> systems if the “back” boxes enbd-extend all it’s drives as one drive)
> fail the entire LVM is down, meaning all the data is gone.
You have to manage to possible problems:
1) hardware failure, not affecting data storage devices (i.e. failing
power supply, failing processor) and not directly causing data loss
2) data storage failure, directly causing data loss (i.e. crashing
harddrive)

Obviously, you don't want to run RAID5 on 200 disks. If you do that, you
could as well invite mr. Murphy to drink a cup of tea at your home.
You should also be aware of the fact that LVM is almost the same as
RAID0: one failing component will bring down your LVM volume. So you'll
have to avoid a failing LVM component.


> So we see these two factors opposing each other: (1) large
> easy-to-manage partition and (2) failover / fault tolerance.
Remembering the things I wrote before, I would suggest something like this:

disk1 \
disk2 |  RAID5
disk3 |  1.5T
disk4 /

disk1 \
disk2 |  RAID5
disk3 |  1.5T
disk4 /

disk1 \
disk2 |  RAID5
disk3 |  1.5T
disk4 /

etc.

And combine those sets using a ENBD client (on a frontend machine).
However, this would require about 67 RAID5 ENBD servers. I'm not really
sure, but I don't think this is really possible using ENBD. It just
isn't designed to fit in such configuration.


> I’m sure you guys have come across similar scenarios and look forward to
> your advice. Maybe it’s not even feasible because network latency would
> prohibit such a large system from running efficiently.
I'm using ENBD for a storage clusters of 3-5 machines, effectively
creating 500G to 2T per cluster (more is possible, but not needed). I've
considered using OpenAFS and other distributed file system solutions,
but those would cause too much overhead in my situation. However,
considering your needs, I think you should be looking for something like
that.

It's possible that Peter (main developer of ENBD) has a completely
different opinion on this topic, so you shouldn't immediately draw your
conclusions on my thoughts. They're just thoughts...

-- Bas