Re: scale of backup and frequency
"C. Chan" <[email protected]>
| Newsgroups | gmane.comp.archivers.amanda.user |
|---|---|
| Message-ID | <[email protected]> |
Also Sprach David Simpson: > Hi, > > That's interesting. > > What is the motivation/main benefits of the Panasas system? (what else did you consider) PanFS has multi-tier storage with metadata going to NVMe, IOPS-bound small files to SATA SSD, and regular files to hard disk. Also seems a bit easier to manage and less pricey compared to GPFS/Spectrum Scale and WekaIO. > Yes inode trawling becoming a big issue. > > ---- > > Am wondering if it might be possible to [sensibly] do full (e.g. monthly) to tape and incremental to disk (e.g. weekly). > > David You don't really need Amanda for that, since its major feature is the scheduler which mixes incrementals and full backups to maximize efficient tape usage. > > ________________________________ > From: C. Chan <[email protected]> > Sent: 24 May 2021 17:06 > To: David Simpson <[email protected]>; [email protected] <[email protected]> > Subject: Re: scale of backup and frequency > > External email to Cardiff University - Take care when replying/opening attachments or links. > Nid ebost mewnol o Brifysgol Caerdydd yw hwn - Cymerwch ofal wrth ateb/agor atodiadau neu ddolenni. > > > > FYI: > > i. Three different backup sets, each around 100TB and growing. > > ii. Fri evening - Mon morning weekend backups, rotated quarterly/every 12 weeks. > Filesystem snapshots done hourly, rotated each day, and daily snapshots, rotated > each week, in place of daily incremental backups. > > iii. The two biggest challenges are the explosion of the total volume of data > and the increase in IOPS limited small file I/O, both mostly due to deep learning. > We are currently using NAS servers with ZFS special devices to store small files > on NVMe SSD drives, but are planning to migrate/consolidate to a Panasas system. > > > Also Sprach David Simpson: > >> Hi all, >> >> Am interested to hear about your Amanda setup. Particularly if you are dealing with an HPC home file system/server, where both size and churn can be issues. >> >> And/or scalable storage. >> >> i) How big is your regular backup? >> ii) How frequent? >> iii) What challenges do you have/face? >> >> I'm currently in the process of thinking about how a new backup regime will look, armed with a new 40 slot tape library and some limited disk storage too. >> >> thanks >> David >> > > > -- > C. Chan <c-chan at uchicago.edu> > GPG Public Key registered at pgp.mit.edu > > -- C. Chan <c-chan at uchicago.edu> GPG Public Key registered at pgp.mit.edu