Handling text files

Hi,

I would like to raise the following issues:

1). The way that zip converts the text file endings (parameter "-l") is
not so useful. The transformation is not idempotent, and it should be.
I propose not to modify zip so that "zip -l" leaves CR LF intact,
instead of transforming it to CR CR LF.

2). If a text file contains the sequence CR CR LF, this is interpreted
by "unzip -a" as two line endings. I think it's better to consider this
as a single line ending.
On the other hand, CR LF LF should still be interpreted as two line
endings.

3). The way that zip distinguishes between binary vs. text files can
never be 100% precise, and yet the current method (which AFAIK is
inherited from PKZIP) guesses wrong many times. I propose to consider
any file that contains any single non-printable control character, a
binary file.

I am not providing patches, because I have to know your opinion about
this first.

**

1). The transformation given by "zip -ll" is idempotent (a Unix text
file remains the same), but the transformation given by "zip -l" is not:
a DOS text file with CR LF's becomes something ugly, with CR CR LF's.

On the other hand, "unzip -a" does the job properly: DOS texts are not
altered by "unzip -a" if they are extracted on DOS/Windows.

My proposition is to change the behavior of zip, in a backwards-
incompatible, but a more useful way. The corresponding help entry would
be changed from:

  -0   store only                   -l   convert LF to CR LF (-ll CR LF to 
LF)

to:

  -0   store only                   -l   end-of-lines to CR LF (-ll to LF)

One could argue that "zip -l" should only be used when archiving Unix
text files, but that's a weak argument. Often, there are mixtures of
DOS and Unix texts, and they have to be handled on a file-by-file basis.
I found zip files containing files with CR LF and CR CR LF line endings
(guess where are they coming from?!). Another experience: I had to make
a zip bundle of zlib-1.2.1 with CR LF endings, and the original tarball
contained LF files, CR LF files, and binary files. I had to use a
separate, 3rd party program to convert all texts uniformly to CR LF.

**

2). This is an issue related to (1).
Currently, "unzip -a" expands CR CR LF to two lines, be it LF LF, or
CR LF CR LF, or CR CR. But this is again something not very useful.
Most likely, CR CR LF comes from programs that make dumb LF -> CR LF
conversions (unfortunately, zip is not the only one).

In the strict interpretation of these two codes on the platforms that
use them both, CR means "jump to the beginning of the same line" and
LF means "jump to the next line (not necessarily at the beginning)".
CR CR .. CR LF should have the same effect as CR LF. On the other hand,
CR LF LF .. LF should have the same effect as CR LF CR LF .. CR LF.
The text editors that are not brain-damaged and make the least of a
recognition between various line endings, use this policy.

So I propose to interpret CR .. CR LF as *one* line ending. This would
make unzip consistent with other programs, and, more importantly, would
provide a fix for the archives created by a dumb "zip -l".

The interpretation of line endings would become:
  LF .. LF	n lines (Unix)
  CR .. CR	n lines (Mac)
  CR LF CR LF	2 lines (DOS/Win)
  CR .. CR LF	1 line  (broken DOS/Win)
  CR LF .. LF	n lines (some flavors on DOS/Win create this)
  etc.

Even Wordpad uses this interpretation, except that it doesn't cope with
CR .. CR sequences alone.

On the other hand, I realize that CR CR LF may come from the
concatenation of Mac and DOS texts, and, in this case, there are indeed
two lines. You might reject proposal (2) because you think that this
interpretation is better; if this is the case, then fine.

**

3). As far as I know, the "binary/text" flag that accompanies every
file in a zip archive is used purely for informational purposes by
PKZIP, but it is used by unzip as an indication whether to apply the
"-a" parameter or not.

I often see incorrect flags. The most common instance is PDF files,
which are binaries if they use Deflate streams, but are often
mis-identified as text.

The current "75% ASCII vs. 25% non-ASCII" threshold was established by
PKZIP, and Info-ZIP also uses it. In my opinion, this is where the
problem resides, because this threshold is not a very good indication.

- If non-printable control characters appear
    iscntrl(ch) && !isspace(ch)
then it's very likely that the file is binary.
It doesn't matter in what proportion do these control characters appear.
If someone insists that 1MB of ASCII characters and a single control
character is a text file, then I'd say it's a broken text file, and
it's not a big deal if this rather isolated case is mis-identified as
binary, and left intact by "unzip -a".

- Counting characters in the range 32-127 vs. 128-255 may work for
people using Latin-based alphabets, but it does not work for the others
who use extended character sets, like ISO-8859 (non-Latin), KOI8, etc.,
or even UTF-8.
I think that the current 75%-25% ratio is a little unfair. I may be
radical to propose not to distinguish at all between 32-127 vs. 128-255,
but I am curious about other opinions, too.

**

In spite of my proposals, I second Greg's plea to release unzip-5.51
before making further important changes. The directory traversal problem
is a security issue that has to be fixed, and the other updates can wait.
I am using unzip-5.51c on a regular basis, and it has worked fine so far.


Best regards,
Cosmin

_________________________________________________________________
Help STOP SPAM with the new MSN 8 and get 2 months FREE*   
http://join.msn.com/?page=dept/bcomm&pgmarket=en-ca&RU=http%3a%2f%2fjoin.msn.com%2f%3fpage%3dmisc%2fspecialoffers%26pgmarket%3den-ca