Re: Using dar for archives to optical discs

On Wed, Jun 21 2023, Denis Corbin wrote:

> [[PGP Signed Part:Undecided]]
>
>
> On 20/06/2023 21:44, John Goerzen wrote:
>> Hi Petr,
>> On Tue, Jun 20 2023, Petr Skoda wrote:
>>
>
> [...]
>
>>
>>> But what I want to suggest to solve is the housekeeping of the dynamically
>>> on-the fly created slices.
>> I would also like to see this.  It would help in my use case of
>> variably-sized external drives.
>
> I'm not sure what you mean by "dynamically on-the fly create slices"... Doesn't
> dar itself, from your filesystem, dynamically creates slices on-the fly of the
> size you asked for?

The problem is when the size of each slice is unpredictable and
variable.  This could occur when:

- Backing up to USB drives of varying capacities

- Backing up to a device with underlying compression (eg, tapes,
  btrfs/ZFS, etc)

- Devices contain non-dar data as well.

What I'm thinking about is: what if dar_split functionality were
integrated into dar, in such a way that it doesn't break slicing and
random access?

Of course, that's not present today because the catalog assumes a
constant slice size (after the first one).  But if the catalog just
saved the size of each slice, it would allow slices to vary in size, and
dar could use the dar_split approach to just end a slice when the target
fills up.

>>> I know Denis has a good explanation why it is not possible currenly and
>>> namely for dar_split (as the system does not
>>> know where it is - it is just a stream of data)
>> If dar itself could sense ENOSPC when writing,
>
> I would rather prefer dar to stay agnostic to the underlying media as much as
> possible. It has proved its value when slices were used to span a backup over
> CDs, then DVDs 20 years ago. At no time dar has had to bother about burning CD-R
> or CD-RW or DVD-R or DVD-RW... but -E and -F option was left for the user to run
> anything that would do that. Today both options can still be used to send/fetch
> slices to/from the cloud for example.

Oh I completely agree.  I'm just suggesting a more flexible slicing
mechanism.

> The catalog grows in memory and does not stores the slice number or number(s)
> file is located in, which may be changed afterward by dar_xform.
>
> But it could be possible after each slice (possible but not simple seen the
> current modular implementation), to dump the current status of the catalogue in
> a separated file (same file format as an isolated catalogue). Though it would
> have all content treated so far, not only the last slice.


I don't *think* we're talking about the same thing here, but I may be
mistaken!

- John