Re: Handling APFS/HFS+

Denis - I thank you for your input.  Questions, and a comment or two, below.

 ---- On Sun, 31 May 2020 09:06:01 -0700 Denis Corbin <[email protected]> wrote ----
 > -----BEGIN PGP SIGNED MESSAGE-----
 > Hash: SHA256
 > 
 > On 31/05/2020 15:42, DiagonalArg wrote:
 > > I'm concerned with backing up an APFS or HFS+ disk to a ZFS
 > > formatted NAS.  I am considering my options, including using `dar`
 > > over `ssh`.
 > 
 > First, there is no dependency between the filesystem you backup and
 > the one you use to store the dar backup onto. I guess this is the same
 > with tar and most other tools.

While I understood this about `dar,` and that's why I'm looking at it, as I mentioned in my OP (below), both rsync and tar had to be modified for HFS+/APFS.  Apple's rsync, of course, would need to mirror to another Mac disk, though there is an rsync+hfsmode project that allows backing up/mirroring to a linux disk using AppleDouble's filename/._filename format.

 > note that since release 2.6.0 you can use sftp and ftp from within
 > dar. This is interesting in particular when restoring a few files from
 > a large archive, as dar will fetch over sftp or ftp the only needed
 > data without requiring any local temporary storage.

TY.

 > > After some research, I see that on MacOS, standard unix tools like 
 > > rsync and tar have been updated to handle the complicated metadata 
 > > structure (Extended Attributes, including variable numbers of 
 > > variable length forks; Finder data; and other metadata).  So they 
 > > will work if backing up to another Apple formatted disk, but not
 > > to "real" unix formats; and the standard unix tools will not be
 > > able to handle the output.  I also see that Apple's NFS has been
 > > updated with the capability of transferring in the "AppleDouble"
 > > format [1] and that their SMB will transfer as something called
 > > "Alternate Data Streams" (ADS) [2],  both of which can be stored to
 > > non-Apple disks. (Though I am not clear what the requirements are
 > > on the receiving end.  If someone knows, please tell me.)
 > > 
 > > So my question is, could someone describe how `dar` backs up 
 > > APFS/HFS+ files, including what kind of format it uses to store
 > > those files in its archive?  AppleDouble?  ADS?  Something else?
 > 
 > Dar supports Extended Attributes from version 1.1.0 June 2002 (!) and
 > has been enhanced in 2006 (release 2.3.0) to support MacOS X file forks,
 > relying on this Extended Attributes feature. 

I did read your docs, and noticed that EA's were covered:
http://dar.linux.free.fr/doc/Features.html

Does that cover everything in these complicated Apple disk structures?  I have also been looking at `restic,` where I found a very illuminating bug report regarding what `restic` can and can't handle when backing up and restoring HFS+ disks.  See the OP, here:
https://github.com/restic/restic/issues/1622

The BackupBouncer software described there was used to test the backup/restore.  If I had a mac machine here, I'd run it to test `dar,` but I am holed up away from home while the plagues and pestilences pass, researching solutions for when I return home.  I won't ask you to do it unless you're interested, but I will certainly like to see the output.

 > "File forks" seems to correspond to what you name ADS ([2])
 
This seems to refer to something different.  That is, Apple was trying to deal with people wanting to back up to a (non Apple) NAS.  They solved this problem initially by offering a modified NFS that saved files on the NAS in AppleDouble format.  (that is two files, filename and ._filename, the latter containing the resource fork and finder data).  The most recent solution applies to SMB and invokes a new way of encoding the various forks and other metadata so they can be stored on a non-Apple system.  They call this ADS, "Alternate Data Streams."  So I was asking if you are using either of these ways of encoding the data, or some other means.

 > Extended Attributes (EA) is just a label associated to some data,
 > which is unstructured from dar point of view as is the content of a
 > file when you consider it at dar or file system level (of course a
 > JPEG file has a structure but for the file system this is just data,
 > as is for dar, as is for file forks).

I understand.

 > So to answer your question, there is not "format" used inside dar
 > archive to store ADS: it is saved as is and restored as is (with a
 > dedicated checksum in the archive). The advantage is that any type of
 > value including "AppleDouble" should be supported.

So this is a misunderstanding.  I'll try to be clearer.  My point is that if I were to store MacOS data on a linux NAS using Apple's modified NFS, I would be able to copy that data to a linux machine and explore and manipulate it there, because it is a simple format, "AppleDouble," consisting of two different data files, easily represented on any filesystem.  If I were to do that same experiment, backing up to the NAS using `dar,` and then looking at the archive by running `dar` from a linux machine, what would I see?  I understand that if I try to restore a file from the archive to a linux FS, I will get error reports; but could I take a look at what's in there the way I could with the ._filename component of AppleDouble?  Could I extract the particular files that may be giving me trouble
  when I try to restore them to the linux FS and save them to a separate dar archive for later handling, without losing any data?  How much control do I have over the contents of the archive *from linux*?

 > > Also, if I were to then try to copy that archive to a linux
 > > machine and explore it with `dar`, what would happen?
 > 
 > The Extended Attributes name has two parts, a name-space and a
 > attribute name separated by a dot. For example "user.test" is an EA
 > name to which is associated an arbitrary long sequence of bytes.
 > 
 > When restoring EA, dar asks the system to create such new EA to the
 > just restored file (or over an already existing file depending on the
 > context). The system may fail/refuse if the support for the
 > corresponding name-space is not present. Dar will then report the
 > failure to the user and continue with the next file or fail depending
 > on options set by the user, but dar will try to restore the most
 > possible things for a give file (data, meta-data, EA and FSA(*))
 > before trying the next one.

Not sure what FSA is?

 > For example "security.selinux" may fail to be restored if the system
 > has no SELinux support, "system.posix_acl_access" may fail to be
 > restored if the filesystem has not been mounted with "acl" support,
 > and so on. I don't recall the EA name used by HFS+ to store file forks
 > but the point is similar.
 > 
 > > Thanks!
 > > 
 > > [1] 
 > > https://en.wikipedia.org/wiki/AppleSingle_and_AppleDouble_formats>
 > > [2] 
 > > https://en.wikipedia.org/wiki/NTFS#Alternate_data_streams_(ADS)
 > > 
 > Cheers,
 > Denis

Yes, I appreciate your input.  Apple certainly hasn't made our lives easy, has it?

/D