Re: 8TB of external storage?

current workflow:
1, 8 laptops with 320g drives hooked to 8 dv cameras, streaming 13g/h for 6
hours.
2. end of day, copy data to file server,
3. from file server copy to 1 of 5 1.5T external drive, put that drive away
(backup)
4. wipe laptops to make room for next days files.
5. use laptops to review/encode data on server.
repeat for 5 days, so more than 320g gets collected on each drive.



new plan:
upgrade laptops to 640g - enough that we don't need to do step 4 (wipe)

add an external drive to each laptop/camera set, stream to both internal and
external.

hook laptops to file server so that encoded files get collected on file
server, but local laptops drives are the primary source of the raw video
files.  external drives are the backups.

externals will be hooked to the server and a 3rd copy made,  but that step
can run in parallel low priority to step 5 (review/encode) because nothing
relies on it, and we have a backup already.  the 3rd copy is really so when
things get behind schedule (they always do) it is easier to work with all
the files in one place.   and having the primary/backup copies moving around
the event has some level of risk.



On Thu, Nov 4, 2010 at 1:11 PM, Timothy J. Wielgos <[email protected]>wrote:

> Holy cow. If you need that kind of speed, you might consider Fibre Channel
> and/or nfs or iscsi over 10gigE.
>
> You can also get a bunch of NICs, preferrably TOE cards, and bond them for
> speed if you're set on a DIY or budget implementation.
>
> We have not had a lot of good results from SATA disks. They tend to be too
> slow and unreliable for our purposes. We only use them for archival purposes
> now - stuff that doesn't require speed.
>
>
>
>  Sent from Yahoo! Mail on Android
>
>  ------------------------------
> * From: * Carl Karsten <[email protected]>;
> * To: * Linux Users Of Northern Illinois (Chicago) - Technical Discussion
> <[email protected]>;
> * Subject: * Re: [LUNI] 8TB of external storage?
> * Sent: * Thu, Nov 4, 2010 3:32:40 AM
>
>   On Wed, Nov 3, 2010 at 9:27 PM, Richard Lynch <[email protected]> wrote:
> > I have no idea what to recommend, but I have spotted a lack of speed
> > requirements here...
> >
> > I suspect that you should have a minimum speed target for the
> > read/write tasks...
> >
> > But I dunno what I'm talking about, as I've never needed anything like
> > 8TB...
>
> good point about speed.  my PyCon video project results in moving 600G
> onto a server each evening, and then doing more work once the data is
> there.  over gig E it takes a few hours, and that basically cuts into
> sleep.  I am looking at changing things around so that I don't have
> that bottleneck
>
> how does one backup 8T?
>
> Or is the point of the box to be a collection of backups?
>
> Or do you rely on raid to deal with disk fail and not worry about the
> other risks.
>
> huh - I wonder how google looks at this problem.
>
> >
> > I accidentally built a 1TB audio server one time, with JBOD, but that
> > was like, an accident, really...
> >
> > I used a case and MOBO from a dumpstered tower and crammed a bunch of
> > hard drives in it and df -h added up to 1TB. :-)
> >
> > It worked quite well over the course of the several years it was in
> > service before the priorities of the client changed... It's in my
> > basement at the moment.
> >
> > On Wed, November 3, 2010 4:19 pm, David Ehle wrote:
> >>
> >> Hello all,
> >>
> >> Long time lurker, wanted to see if anyone has any experience and
> >> suggestions regarding medium-large storage solutions.
> >>
> >> I need to upgrade and consolidate our storage backups.
> >>
> >> I would like to start with 6-8TBs
> >>
> >> I want it rack mountable.
> >>
> >> I want to run RAID5 or RAID6, and be able to add more/larger drives to
> >> the
> >> array in the future to expand it.
> >>
> >> It must speak sweetly to linux.
> >>
> >> I've looked at NAS solutions like products by Drobo,QNAP,Thecus and
> >> they
> >> seem to be about the most cost effective... but now that linux
> >> supports
> >> expanding software RAID on the fly, I'm not really sure they offer
> >> anything extra and add an extra layer of potential failure / lockin /
> >> interface.
> >>
> >> Dell/IBM solutions just cost too much and are overengineered for the
> >> task
> >> at hand.
> >>
> >> Buying a new system with 8 hotswap sas/sata bays is an option, but
> >> expensive. (compared to a NAS black box)
> >>
> >> Buying an external enclosure is cost competitive to Drobo/Qnap/Thecus
> >> black box solutions, but I would want esata or SAS connectivity... and
> >> while I've done eSATA I have not done much work with external SAS. I'm
> >> not
> >> sure if how the drives will be presented to the OS if enclosure is
> >> connected via 1 cable.  iSCSI is also an option I guess but then
> >> wouldn't
> >> I need a "smart" box rather than JBOD backplane?  If going this route,
> >> would I go with just some vendors "a esata/SAS card" or would I need
> >> some
> >> sort of hardware RAID card?
> >>
> >> Anyone else delt with a similar situation? Any insite or advice?
> >>
> >> I've thought about some of the Storage distributions like
> >> Openfiler/OpenNas - but I don't really feel like learning BSD for this
> >> project, and I think opensolaris based projects are going to die do to
> >> lack of support now that Oracle has abandoned OpenSolaris (bye bye ZFS
> >> :P)
> >>
> >> Any experience with Drobo/Qnap/Thecus?  Which is most linux friendly?
> >> Which can you take the disks and attach them to something else if the
> >> hardware dies? (Drobo is out because of this, Qnap uses "custom" ext3
> >> version but is readable from bootable ISO, thecus I'm not sure about)
> >>
> >> Anywy thanks for listening to me think out loud... what's your input?
> >>
> >> Thanks!
> >>
> >> David.
> >>
> >>
> >> --
> >> Linux Users Of Northern Illinois (Chicago) - Technical Discussion
> >> http://luni.org/mailman/listinfo/luni
> >>
> >
> >
> > --
> > brain cancer update:
> > http://richardlynch.blogspot.com/search/label/brain%20tumor
> > Donate:
> >
> https://www.paypal.com/cgi-bin/webscr?cmd=_s-xclick&hosted_button_id=FS9NLTNEEKWBE
> >
> >
> > --
> > Linux Users Of Northern Illinois (Chicago) - Technical Discussion
> > http://luni.org/mailman/listinfo/luni
> >
>
>
>
> --
> Carl K
>
> --
> Linux Users Of Northern Illinois (Chicago) - Technical Discussion
> http://luni.org/mailman/listinfo/luni
>



-- 
Carl K

-- 
Linux Users Of Northern Illinois (Chicago) - Technical Discussion 
http://luni.org/mailman/listinfo/luni