Re: Handling APFS/HFS+

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 31/05/2020 15:42, DiagonalArg wrote:
> I'm concerned with backing up an APFS or HFS+ disk to a ZFS
> formatted NAS.  I am considering my options, including using `dar`
> over `ssh`.

First, there is no dependency between the filesystem you backup and
the one you use to store the dar backup onto. I guess this is the same
with tar and most other tools.

note that since release 2.6.0 you can use sftp and ftp from within
dar. This is interesting in particular when restoring a few files from
a large archive, as dar will fetch over sftp or ftp the only needed
data without requiring any local temporary storage.

> 
> After some research, I see that on MacOS, standard unix tools like 
> rsync and tar have been updated to handle the complicated metadata 
> structure (Extended Attributes, including variable numbers of 
> variable length forks; Finder data; and other metadata).  So they 
> will work if backing up to another Apple formatted disk, but not
> to "real" unix formats; and the standard unix tools will not be
> able to handle the output.  I also see that Apple's NFS has been
> updated with the capability of transferring in the "AppleDouble"
> format [1] and that their SMB will transfer as something called
> "Alternate Data Streams" (ADS) [2],  both of which can be stored to
> non-Apple disks. (Though I am not clear what the requirements are
> on the receiving end.  If someone knows, please tell me.)
> 
> So my question is, could someone describe how `dar` backs up 
> APFS/HFS+ files, including what kind of format it uses to store
> those files in its archive?  AppleDouble?  ADS?  Something else?

Dar supports Extended Attributes from version 1.1.0 June 2002 (!) and
has been enhanced in 2006 (release 2.3.0) to support MacOS X file forks,
relying on this Extended Attributes feature. "File forks" seems to
correspond to what you name ADS ([2])

Extended Attributes (EA) is just a label associated to some data,
which is unstructured from dar point of view as is the content of a
file when you consider it at dar or file system level (of course a
JPEG file has a structure but for the file system this is just data,
as is for dar, as is for file forks).

So to answer your question, there is not "format" used inside dar
archive to store ADS: it is saved as is and restored as is (with a
dedicated checksum in the archive). The advantage is that any type of
value including "AppleDouble" should be supported.

> 
> Also, if I were to then try to copy that archive to a linux
> machine and explore it with `dar`, what would happen?

The Extended Attributes name has two parts, a name-space and a
attribute name separated by a dot. For example "user.test" is an EA
name to which is associated an arbitrary long sequence of bytes.

When restoring EA, dar asks the system to create such new EA to the
just restored file (or over an already existing file depending on the
context). The system may fail/refuse if the support for the
corresponding name-space is not present. Dar will then report the
failure to the user and continue with the next file or fail depending
on options set by the user, but dar will try to restore the most
possible things for a give file (data, meta-data, EA and FSA(*))
before trying the next one.

For example "security.selinux" may fail to be restored if the system
has no SELinux support, "system.posix_acl_access" may fail to be
restored if the filesystem has not been mounted with "acl" support,
and so on. I don't recall the EA name used by HFS+ to store file forks
but the point is similar.

> 
> Thanks!
> 
> [1] 
> https://en.wikipedia.org/wiki/AppleSingle_and_AppleDouble_formats>
> [2] 
> https://en.wikipedia.org/wiki/NTFS#Alternate_data_streams_(ADS)
> 
> 

Cheers,
Denis

(*)File system Specific Attributes
-----BEGIN PGP SIGNATURE-----

iQIzBAEBCAAdFiEEOzEprx3d76WjfYGPCDGwvQPYsYIFAl7T1eEACgkQCDGwvQPY
sYIqvg/+J9ZNT6xDUPxJckTc8DkzbxPiYcI4hdSchezydZdS1igIS8IvHyovkF0w
g/ZK+Yi8rfixQ2lZx4bqTT0YfVmbo/I0eQmJgP7gpqKDiT+6bYVQsQwPwwctkVFE
a4gFvKMHYmdbmfsuqpOYU15MmA4rRl/7EkJbbUxMHYzoo3AzLe1we6LJV+NSU2HU
KqAdFcEnV1/tNAlTbhLJ0tw323YcWsaeHSrJbQXCHvs5HJOwL6I+2s9GXI8A9oK5
acKvE/OyMph3haIZapUq00Qb8PMD/wAZ+uLN+0QUkwRLcQZfXrz4wg0oFWpj+cNZ
v5tFa3JvWZ1JLCky6UyHfIpdZpYtRqdLnQ25c9XxDzrZmBDu9yxRZ84iBU/QORcI
8SgTDz/Mb9V/iM+kJ/zIZLFwi0mo5H9wpke+qe25ACo2TRrcmYzhtm3UGxGDnmV6
lxrhLWelHo96R74aGlQONY379bPcIlIsmMv0Ycs2sF1B8fYVGNm48FJMBShNiVum
exz9WxjyD4qsgjKNUcH6VOqQn4Xd4RmPMTQugBCqlQcmybq+iARxxuWmwRxykPsn
H6rdUGEWJ8DKEnAJ+E8e+g84GFypMXp+jTjCVcuTbnuVXei+E/G1yseoJOMg82dd
69/16N/5jhXl1VcklYiu8gHC/VDS8i7vQFhaQAVcko/p2RJM5ik=
=Gte/
-----END PGP SIGNATURE-----