Re: Handling APFS/HFS+

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 01/06/2020 12:23, DiagonalArg wrote:
> Denis - I thank you for your input.  Questions, and a comment or
> two, below.
> 
> ---- On Sun, 31 May 2020 09:06:01 -0700 Denis Corbin 
> <[email protected]> wrote ----
>> 
>> On 31/05/2020 15:42, DiagonalArg wrote:
>>> I'm concerned with backing up an APFS or HFS+ disk to a ZFS 
>>> formatted NAS.  I am considering my options, including using 
>>> `dar` over `ssh`.
>> 

[...]

>>> I also see that Apple's NFS has been updated with the
>>> capability of transferring in the "AppleDouble" format [1] and
>>> that their SMB will transfer as something called "Alternate
>>> Data Streams" (ADS) [2],  both of which can be stored to
>>> non-Apple disks. (Though I am not clear what the requirements
>>> are on the receiving end.  If someone knows, please tell me.)
>>> 
>>> So my question is, could someone describe how `dar` backs up 
>>> APFS/HFS+ files, including what kind of format it uses to store
>>>  those files in its archive?  AppleDouble?  ADS?  Something
>>> else?
>> 
>> Dar supports Extended Attributes from version 1.1.0 June 2002 (!)
>>  and has been enhanced in 2006 (release 2.3.0) to support MacOS X
>>  file forks, relying on this Extended Attributes feature.
> 
> I did read your docs, and noticed that EA's were covered: 
> http://dar.linux.free.fr/doc/Features.html

Good, not everybody does that before asking...

> 
> Does that cover everything in these complicated Apple disk 
> structures?  I have also been looking at `restic,` where I found a
>  very illuminating bug report regarding what `restic` can and can't
>  handle when backing up and restoring HFS+ disks.  See the OP,
> here: https://github.com/restic/restic/issues/1622

Well, according to what restic checks, dar should fail on:
- - finder-locks
- - ACL
- - some bsd-flags

for bsd-flags dar handles them by mean of FSA (read further for
details), but today I guess only the nodump flag is handled by dar. That
should not be very complicated to add some other ones if needed the
data structures are already there to
handle them.

> 
> The BackupBouncer software described there was used to test the 
> backup/restore.  If I had a mac machine here, I'd run it to test 
> `dar,` but I am holed up away from home while the plagues and 
> pestilences pass, researching solutions for when I return home.  I
>  won't ask you to do it unless you're interested, but I will
> certainly like to see the output.

If I had a mac machine I would have already tested it! Anyway, I would
still be interested to know the test result of dar with restic on MacOS
when you'll be back home.

> 
>> "File forks" seems to correspond to what you name ADS ([2])
> 
> This seems to refer to something different.  That is, Apple was 
> trying to deal with people wanting to back up to a (non Apple) NAS.
>  They solved this problem initially by offering a modified NFS that
>  saved files on the NAS in AppleDouble format.  (that is two files,
>  filename and ._filename, the latter containing the resource fork
> and finder data).  The most recent solution applies to SMB and
> invokes a new way of encoding the various forks and other metadata
> so they can be stored on a non-Apple system.  They call this ADS,
> "Alternate Data Streams."  So I was asking if you are using either
> of these ways of encoding the data, or some other means.

the short answer is no.

Though if saving a file system having AppleDouble format, as I
understand it to only add plain files, dar will save and backup them
properly. While if running dar from the Mac NFS client host of this NAS,
I assume those AppleDouble (or more recent solution) would be exposed as
if it was a normal file system supporting file forks, finder data, bsd
attributes and so on, and here too, dar will do the most it can as
described above for HFS+, there will not be ._filename exposed by the
NFS client but file forks that dar will save as such with the file's
data and other metadata (with limitation mentioned above).

[...]

> 
>> So to answer your question, there is not "format" used inside dar
>>  archive to store ADS: it is saved as is and restored as is (with
>> a dedicated checksum in the archive). The advantage is that any
>> type of value including "AppleDouble" should be supported.
> 
> So this is a misunderstanding.  I'll try to be clearer.  My point
> is that if I were to store MacOS data on a linux NAS using Apple's
>  modified NFS, I would be able to copy that data to a linux machine
>  and explore and manipulate it there, because it is a simple
> format, "AppleDouble," consisting of two different data files,
> easily represented on any filesystem.

You could thus backup and restore this properly using dar from and to
this NAS. But dar will only see ._filename entries and will not try to
understand the format used in them.

Thus, you will not have dar converting those AppleDouble to libdar
internal structure holding the file forks and BSD attributes. They will
be saved as and restored as ._filename, even if the target file system
has full support for file forks and other attributes.

> If I were to do that same experiment, backing up to the NAS using 
> `dar,` and then looking at the archive by running `dar` from a
> linux machine, what would I see?

you will see the content of the NAS filesystem, thus a set of files and
directories, including the ._filenames.

> I understand that if I try to restore a file from the archive to a
>  linux FS, I will get error reports;

No, if you backup the NAS filesystem where file forks and other MacOS
stuff is stored as AppleDouble, you will restore AppleDouble which, as
far as I understand are plain files, so this should not cause any
specific problem.

You will get some errors if you backup from a Mac OS the data stored on
local disk or on Apple enhanced NFS file system and try to restore to a
Linux machine which does not have birthtime, file forks and the rest.

> but could I take a look at what's in there the way I could with
> the ._filename component of AppleDouble?

Yes, you will see the ._filenames, as they are present on the NAS. For
the NAS, the ._filename are just pure normal files as is for dar when
run on the NAS. The fact the Apple NFS client add those file to
associate additional meta data to each file is only used/understood as
such by the Apple NFS client.

> Could I extract the particular files that may be giving me trouble
>  when I try to restore them to the linux FS and save them to a 
> separate dar archive for later handling, without losing any data?

if you have cannot restore a file and want to keep as much information
of it in a separated archive, you should use the filtering feature (see
merging feature) that let you create an archive from a subset of a
bigger one. This is done without restoring the files/directories so you
do not depend on the operating system to support all data and meta-data
associated to that file(s).

> How much control do I have over the contents of the archive *from
> li nux*?

In short, you can define which file and directory to save, which EA and
FSA to take into account, which file to save the data to the archive or
just store as existing file (context: snapshot and differential backups)
.

For a detailed answer, as you have found the documentation ;^) you
should easily get it.

> 
>>> Also, if I were to then try to copy that archive to a linux 
>>> machine and explore it with `dar`, what would happen?
>> 
>> The Extended Attributes name has two parts, a name-space and a 
>> attribute name separated by a dot. For example "user.test" is an
>> EA name to which is associated an arbitrary long sequence of
>> bytes.
>> 
>> When restoring EA, dar asks the system to create such new EA to
>> the just restored file (or over an already existing file
>> depending on the context). The system may fail/refuse if the
>> support for the corresponding name-space is not present. Dar will
>> then report the failure to the user and continue with the next
>> file or fail depending on options set by the user, but dar will
>> try to restore the most possible things for a give file (data,
>> meta-data, EA and FSA(*)) before trying the next one.
> 
> Not sure what FSA is?

FSA = Filesystem Specific Attributes.

for details see
https://en.wikipedia.org/wiki/Chattr
and
https://www.freebsd.org/cgi/man.cgi?query=chflags&sektion=1

Note that all lsattr/chattr attributes are taken into account this way,
this is not the case for the bsd chflags (these are the bsd flags
mentioned above about restic), but the current implementation can easily
be extended if needed, as mentioned previously.

[...]

> 
> Yes, I appreciate your input.  Apple certainly hasn't made our
> lives easy, has it?

Well, I bet they did, by taking BSD as root of MacOS X, which advantaged
Unix-es systems (and Linux in particular) against Windows proprietary
systems. This simplified a lot of thing and put back openness in the
center of the game... See now, even Microsoft now likes Unix! :-)

> 
> /D
> 

-----BEGIN PGP SIGNATURE-----

iQIzBAEBCAAdFiEEOzEprx3d76WjfYGPCDGwvQPYsYIFAl7VUe4ACgkQCDGwvQPY
sYL5VA/8DmDLuHN98mvhMGCftV3lx3igkkePkNS2cNUtaRJ5uSLD2+PIlEHAs8h4
LXQQ6U/UnDMYZ1nyetDqA5tgBxUCuR22qTu+n2ismQguvcXNe4GlOf2S7SY/fA5H
E+UMj8+cjpCckXwT1MAhvHEFIv5qRBHBmvjTUo5ESgcb102nQcIq8fCBeMh9Dzg3
YBrE7gZRBwz/maptPu43MzmInxEGqC632MPa75J9St5q2BiPwZnQCJep5YynINmQ
I6//Hxdp6ser/B8jEBiqLAQ4BF9lAM4rbAXh+BFaazr0TGxMWCZIi7FtY1QnMFov
gCSaOSYjbWQTY06NBQloza5VCrhwKQBuhc93SvPfzZYOL9+580od46qdbydKe8+W
xKyuIMjk+mp3SRYYiCEevqBzz1U4DH3cVcPNtxxRvTTrvBFv+oN2rBCF+WoQvJ0V
kbOQdTEKg0oJRseajCBkSS38oXdaAO48XPLoPdyG3flot62tgWLXo7avvMANsVb+
ewEDRxeqLG+DhexVUpzW92dRn3eYcVZym3C8gT7aFFZ4No7ilisoHHtySBcDBnVF
WD91iT/NfyOiyo7Heb57v/2BbIu0xO6P4H7vJFsVn0S/q14BN2Dop4Xll1DApeLd
JGkXgX6Oh1xA/HPUhVQHDW2yemDB3pW55qGnaHB+nyzbrJolQ/c=
=VvBQ
-----END PGP SIGNATURE-----