Re: compressed release distribution formats (was: GNU Coding Standards, automake, and the recent xz-utils backdoor)

Eric Blake wrote:
> [adding in coreutils, for some history]
>
> [...]
>
> At any rate, it is now obvious (in hindsight) that zstd has a much
> larger development team than xz, which may alter the ability of zstd
> being backdoored in the same way that xz was, merely by social
> engineering of a lone maintainer.
>   

That just means that a cracker group needs to plant a mole in a larger 
team, which was effectively the goal of the sockpuppet campaign against 
the xz-utils maintainer, except that the cracker's sockpuppet was the 
second member of a two-member team.  I see no real difference here.

I would argue that GNU software should be consistently available in at 
least one format that can be unpacked using only tools that are also 
provided by the GNU project.  I believe that currently means "gzip", 
unfortunately.  We should probably look to adopt another one; perhaps 
the lzip maintainer might be interested?

> It is also obvious that having GNU distributions available through
> only a SINGLE compression format, when that format may be vulnerable,
>   
The xz format is not vulnerable, or at least has not been shown to be so 
in the sense of security risks, and only xz-utils was backdoored.  Nor 
is there only one implementation:  7-zip can also handle xz files.
> is a dis-service to users when it is not much harder to provide
> tarballs in multiple formats.  Having multiple tarballs as the
> recommendation can at least let us automate that each of the tarballs
> has the same contents,
Agreed.  In fact, if multiple formats can be produced concurrently, we 
could validate that the compressed tarballs are actually identical.  
(Generate using `tar -cf - [...] | tee >(compress1 >[...].tar.comp1) | 
tee >(compress2 >[...].tar.comp2) | gzip -9 >[...].tar.gz` if you do not 
want to actually write the uncompressed tarball to the disk.)  But if 
tarlz is to be used to write the lzipped tarball, you probably want to 
settle for "same file contents", since tarlz only supports pax format 
and we may want to allow older tar programs to unpack GNU releases.
>  although it won't make it any more obvious
> whether those contents match what was in git (which was how the xz
> backdoor got past so many people in the first place).
This is another widespread misunderstanding---almost all of the xz 
backdoor was hidden in plain sight (admittedly, compressed and/or 
encrypted) *in* the Git repository.  The only piece of the backdoor not 
found in Git was the modified build-to-host.m4.  The xz-utils project's 
standard practice had been to /not/ commit imported m4 files, but to 
bring them in when preparing release tarballs.  The cracker simply 
rolled the "key" to the dropper into the release tarball.  I still have 
not seen whether the configure script in the release tarball was built 
with the modified build-to-host.m4 or if the crackers were depending on 
distribution packagers to regenerate configure.

Again, everything present in both Git and the release tarball /was/ 
/identical/.  There were no mismatches, only files added to the release 
that are not in the repository, and that are /expected/ to be added to a 
release.


-- Jacob