Re: Using dar for archives to optical discs

On 22/06/2023 03:43, John Goerzen wrote:
> 
> On Wed, Jun 21 2023, Denis Corbin wrote:
> 
>> [[PGP Signed Part:Undecided]]
>>
>>
>> On 20/06/2023 21:44, John Goerzen wrote:
>>> Hi Petr,
>>> On Tue, Jun 20 2023, Petr Skoda wrote:
>>>
>>
>> [...]
>>
>>>
>>>> But what I want to suggest to solve is the housekeeping of the dynamically
>>>> on-the fly created slices.
>>> I would also like to see this.  It would help in my use case of
>>> variably-sized external drives.
>>
>> I'm not sure what you mean by "dynamically on-the fly create slices"... Doesn't
>> dar itself, from your filesystem, dynamically creates slices on-the fly of the
>> size you asked for?
> 
> The problem is when the size of each slice is unpredictable and
> variable.  This could occur when:
> 
> - Backing up to USB drives of varying capacities

this can be addresses by using as slice size, the greatest common 
divisor amount USB key sizes. For example, keys set of 15 GB, 12 GB and 
9 GB would lead to chose 3 GB for the slice size and have 5, 4 slices 
and 3 slices per USB keys depending on the key size.

> 
> - Backing up to a device with underlying compression (eg, tapes,
>    btrfs/ZFS, etc)

why not relying on dar's ability to compress still keeping fixed slice 
sizes?

Even if there is an hypothetical time penalty (though, the tape you 
mentioned do seem to be very slow), it would worth it, no? More over 
considering that since release 2.7.0 you can use multi-threading 
compression within dar (OK, this is not very helpful if you only have 
only small files, as dar compresses per file).

> 
> - Devices contain non-dar data as well.

Why not put them all in the first(s) devices and use dar's -S option to 
adjust the first slice size accordingly to the remaining space on the 
last device partially filled with non-dar data?

And this can be repeated alternating non-dar data followed by dar 
backup... over and over

> 
> What I'm thinking about is: what if dar_split functionality were
> integrated into dar, in such a way that it doesn't break slicing and
> random access?

it would require storing the size of all slices in order to fetch a 
particular data/file from the backup... technically it could be possible 
to add a new class at the layer where sar and trivial_sar act, class 
doing precisely that: recording the list of offset where writing failed 
(hitting the end of a tape) and appending such list at the end of the 
backup process.

But such list would just be useless for sequential reading as it would 
be read at the end of the backup... and using direct access read would 
mean that the devices used have the ability to store a filesystem, thus 
the size of each slice would no more be "unpredictable and variable" 
seen the previous proposal of using variable sized USB keys.

But yes, it would be interesting to have such list of slices size within 
an isolated catalogue. This way, restoring a particular file or set of 
file with the help of an isolated catalogue, dar would only read the 
needed slices (sequentially or not), which is not possible today with 
dar_split.

-> feel free to add a feature request! :)

> 
> Of course, that's not present today because the catalog assumes a
> constant slice size (after the first one).  But if the catalog just
> saved the size of each slice, it would allow slices to vary in size, and
> dar could use the dar_split approach to just end a slice when the target
> fills up.

yes... but this would not be stored in the catalogue, the catalogue does 
not know about slicing, but anyway, the information should be stored 
somewhere in the archive, I agree.

> 
>>>> I know Denis has a good explanation why it is not possible currenly and
>>>> namely for dar_split (as the system does not
>>>> know where it is - it is just a stream of data)
>>> If dar itself could sense ENOSPC when writing,
>>
>> I would rather prefer dar to stay agnostic to the underlying media as much as
>> possible. It has proved its value when slices were used to span a backup over
>> CDs, then DVDs 20 years ago. At no time dar has had to bother about burning CD-R
>> or CD-RW or DVD-R or DVD-RW... but -E and -F option was left for the user to run
>> anything that would do that. Today both options can still be used to send/fetch
>> slices to/from the cloud for example.
> 
> Oh I completely agree.  I'm just suggesting a more flexible slicing
> mechanism.

OK

[...]

> 
> - John
>