Re: Huge archive and last slice

Dear Petr,

I have to highlight that I am on the very beginning of dar + tape library solution, and most likely there will be much more for me to learn from you than the other way around.

Allow me to describe the use case, idea and expectations.

The infrastructure and configuration at a glance (needed for understanding the context):
* the data is our users scientific data, mix and match of everything they use on our HPC clusters
* tape library "TL-A" that holds backup of data
* tape library "TL-B" that I am migrating data from TL-A
* tape libraries are on the backend of gpfs+hsm (hsm stands for: hirarchical storage management) solution, particulary TL-B is configured as tape storage pool for HSM
* data is automatically migrated from disk pools to tapes based on configured policy (file size, atime etc)

Use case:
I am migrating data from backup from TL-A to gpfs filesystem with HSM. It's somwhere around 700TB. Before data is migrated to tapes, they are saved from TL-A to gpfs disk pool.
Now, my issue is that users - as they always tend to - didn't respect the regulations and put plenty of small files (<100MB) into the system, which causes two issues: a) gpfs disk storage pool gets bloated with small files and cache size is reduced b) migrations, recalling and handling such data on tapes is a pain. They get spread across multiple tapes, fragmentation occurs, databases get large and overall performance of systems significantly drops.

My task is to make a cleanup by packaging data. For example I have a user that has ~350TB of data, both nice (>1GB files) and not nice. During migration from TL-A to gpfs disk pool, I monitor the directory structure and identify directories that has a lot of small files. Upon user acknowledgement I am packaging such directories with dar.
The idea is to create multiple dar archives in user directory, but have a single database for each user. Dar archives are meant to be smaller than 1TB, therefore my idea is that user can use `dar_manager` to discover in which archive his file resides and use that single archive. By doing so, I expect that discovering files will be done with local database without touching archives residing on tape library. Without `dar_manager` in order to read the archive contents, entire archive would be recalled from tape pool to disk pool. Therefore it's a win-win for me - I'm saving resources by avoidance of unnecessary recalls and I am happy with my data structure residing on tapes.

At this moment I am in very early stage - I've identified some directories and packed one of them with dar.
I do have my own wrapper for dar written in bash, because dar is large program, with plenty of functions and not obvious concepts at first glance, and I'll have to be a 24/7 support for my users to explain them how to use it properly.
Due to lack of time progress is very slow - perhaps in this week I'll start building up database and perform tests if solution works as I expect it to work.
It has to be mentioned that this system is an "archive" solution, so by design these data is "cold" - in other words not used often.

Regarding the compression, yes that could be a rough topic, I haven't tested it the way you did. I am solely relying on tape drives compression methods.

I'd be happy to share my experiences as they develop and dar otions I'm using. I'm just not sure if this mailing list is the right place to do so?

Best regards,
Patryk.

Wroclaw Centre for Networking and Supercomputing

On 23/12/18 02:44PM, Petr Skoda wrote:
[-- Type: text/plain; charset=utf-8, Encoding: quoted-printable, Size: 2.2K --]
> 
> Dear Patryk,
> 
> 
> 
> 
> you have mentioned the tape library and cache system in relation to using 
> dar.
> 
> 
> Would you mind to describe us more about your setup, configuration of dar ,
> parameters of its command line or perhaps scripts you are using for handling
> the large amount of data ?
> 
> 
> 
> 
> Until recently I supposed I was the only one who tried to use dar together
> with LTO ULTRIUM drives. I have made a lot of experiements but so far I am
> not fully satisfied with the dar functionality on tapes. e.g. the sequential
> mode recovery from slices, whether to use fixed slices or  dar_split, if it
> makes sense the parameters for limiting data flow etc. (dar_split -b -r) 
> 
> 
> I am also not sure whether it is better to use the LTO HW compression or not
> ... . I have made a lot of acoustic tests (hearing the LTO drive in open 
> computer on my desk during various operations, and it seems the answer is 
> not easy (to compress or not) .... Even the large buffers done by mbuffer 
> will not prevent shoe-shining of the tape .
> 
> So I am still looking for the best practice of using LTO  (drive or library)
> together with dar to assure optimal (easy, fast) storage of tens of TB from
> big arrays (well based on zfs RAIDs).
> 
> 
> So I hope that not only myself would benefit a lot from you optimal big data
> handling use case ....
> 
> 
> 
> 
> Thanks a lot
> 
> 
> 
> 
> 
> Petr Skoda
> 
> Astronomical Institute of the Czech Academy of Sciences.
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> ---------- Původní e-mail ----------
> Od: Patryk Bełzak <[email protected]>
> Komu: need help compiling or using dar ? This mailing list is for you. <dar-
> [email protected]>
> Datum: 13. 12. 2023 9:29:51
> Předmět: Re: [Dar-support] Huge archive and last slice
> "I guess it's also worth to mention `dar_manager`.
> I'm also dealing with PB of scentific data in Hierarchical Storage
> Management solution, and I'm considering scenario where I'll build a
> database on some nfs (so it could be accessed by all systems I need, and 
> getting some sort of high availability).
> By doing so, searching for files and extracting them wouldn't require to 
> recall large amounts of data from tape library to cache filesystem.
> 
> "

[-- Alternative Type #1: text/html; charset=utf-8, Encoding: quoted-printable, Size: 2.6K --]