Re: scale of backup and frequency

Yes. With the holding area, no benefit of going to disk for the incrementals. Might as well go to tape also.

Currently using an rsync scripted backup for cluster NFS home but limited dirs. Tape library to be commissioned soon.

Panasas - will you be writing to that from sequencers or instruments and using it for processing on compute cluster? Interested to know about the admin effort point, also something they mention on their site - perhaps more like effort of looking after an isilon? . Snapshots presumably on it? Are there lots of licenses needed for features?
________________________________
From: C. Chan <[email protected]>
Sent: 25 May 2021 18:30
To: David Simpson <[email protected]>; [email protected] <[email protected]>
Subject: Re: scale of backup and frequency

Also Sprach David Simpson:

> Hi,
>
> That's interesting.
>
> What is the motivation/main benefits of the Panasas system? (what else did you consider)

PanFS has multi-tier storage with metadata going to NVMe, IOPS-bound small files to SATA
SSD, and regular files to hard disk.  Also seems a bit easier to manage and less pricey
compared to GPFS/Spectrum Scale and WekaIO.


> Yes inode trawling becoming a big issue.
>
> ----
>
> Am wondering if it might be possible to [sensibly] do full (e.g. monthly) to tape and incremental to disk (e.g. weekly).
>
> David

You don't really need Amanda for that, since its major feature is the scheduler which
mixes incrementals and full backups to maximize efficient tape usage.

>
> ________________________________
> From: C. Chan <[email protected]>
> Sent: 24 May 2021 17:06
> To: David Simpson <[email protected]>; [email protected] <[email protected]>
> Subject: Re: scale of backup and frequency
>
> External email to Cardiff University - Take care when replying/opening attachments or links.
> Nid ebost mewnol o Brifysgol Caerdydd yw hwn - Cymerwch ofal wrth ateb/agor atodiadau neu ddolenni.
>
>
>
> FYI:
>
> i. Three different backup sets, each around 100TB and growing.
>
> ii. Fri evening - Mon morning weekend backups, rotated quarterly/every 12 weeks.
>     Filesystem snapshots done hourly, rotated each day, and daily snapshots, rotated
>     each week, in place of daily incremental backups.
>
> iii.  The two biggest challenges are the explosion of the total volume of data
>       and the increase in IOPS limited small file I/O, both mostly due to deep learning.
>       We are currently using NAS servers with ZFS special devices to store small files
>       on NVMe SSD drives, but are planning to migrate/consolidate to a Panasas system.
>
>
> Also Sprach David Simpson:
>
>> Hi all,
>>
>> Am interested to hear about your Amanda setup. Particularly if you are dealing with an HPC home file system/server, where both size and churn can be issues.
>>
>> And/or scalable storage.
>>
>> i) How big is your regular backup?
>> ii) How frequent?
>> iii) What challenges do you have/face?
>>
>> I'm currently in the process of thinking about how a new backup regime will look, armed with a new 40 slot tape library and some limited disk storage too.
>>
>> thanks
>> David
>>
>
>
> --
> C. Chan <c-chan at uchicago.edu>
> GPG Public Key registered at pgp.mit.edu
>
>


--
C. Chan <c-chan at uchicago.edu>
GPG Public Key registered at pgp.mit.edu