Ftp-WG: NLST vs MLSD

Alun Jones <[email protected]>, Sun, 13 Oct 2002 16:53:08 -0500:
: At 01:26 PM 10/13/2002, Frank da Cruz wrote:
: >That the MLSD spec does not permit filenames or mixtures of paths and
: >filenames in its argument; thus the user is not allowed to send this
: >argument to the server in an MLSD command.  Note that I don't advocate
: >changing MLSD in this respect.  But from the user's point of view, a
: >command that once worked will no longer work and this will cause confusion,
: 
: Why?  You've given no indication as to why MGET - the "command that once 
: worked" should use anything other than NLST.  Hence, MGET will work exactly 
: the same way it does now.
: 
I guess I need to spell it out:

 1. MGET works by getting a list of files from the server and then
    RETR'ing each file.

 2. New clients will want to use MLSD because of its enhanced list format.
    
 3. For backwards compatibility, clients will want to continue to have
    an MGET command with the same syntax as before.

What is the new client supposed to do with an argument like "pics/m*.jpg"?

 1. It can't include it as an MLSD argument because MLSD accepts only
    directory names.

 2. But how is the client supposed to know whether it's a directory name
    when it's in the syntax of the server's file system?

      pics/m*.jpg      (UNIX)
      pics:m+.jpg      (AOS/VS)
      pics\m*.jpg      (DOS)
      pics>m*.jpg      (VOS)
      m* jpg 197       (CMS)
      [.pics]m*.jpg    (VMS) 
      pics:m*.jpg      (also VMS)
      <.pics>m*.jpg    (TOPS-20) 
      [123,456]m*.jpg  (TOPS-10)

: >An operation that was possible before is suddenly impossible.
: 
: Only if you insist that MGET must use MLSD.  Why not stick with NLST, as 
: you are today?  MLST / MLSD replaces only one (inappropriate) use of LIST, 
: not NLST.
: 
If MLSD is not for use with MGET then what is it for?  It is not a LIST
replacement.  LIST is for people, MLSD is for machines.

I don't insist that MGET use only MLSD -- please go back and read:

  http://www.columbia.edu/kermit/newftp.html

Sometimes NLST is better, sometimes MLSD is better.  Thus a client can try
to choose the best FTP command in each situation.  But it can't always be
right when the choice depends on the server's file system syntax, which the
client can not be expected to know unless we reduce the world to Unix and
Windows.  Thus the user must be exposed to the details and tradeoffs of NLST
versus MLSD.

: >An operation that was possible before is suddenly impossible.
:
Anyway you took this statement out of context.  It applies to the situation
in which the client expects the server to expand wildcards (as virtually
every client does) but the server interprets RFC959 strictly and does not
accpet wildcards.

: >If a text file is sent in binary mode to a radically different platform,
: >the result is garbage (and vice-versa, of course).
: 
: Incorrect.  If a text file is sent in binary mode, the result may be a file 
: where the line-ends are not in the local format, but the file is far from 
: being garbage.  Tools exist even on DOS - very simple tools - to convert 
: "Mac" and "Unix" style line ends to "DOS" style, and vice-versa.
: 
If you restrict yourself to Unix and DOS that might be true.  Once you
allow for other platforms you find:

 . Variable-length record files with embedded record-length fields in
   a variety of formats (binary, ASCII numeric, packed decimal) and 
   having different lengths (2 bytes, 4 bytes, ...)

 . Fixed-length records.

 . EBCDIC instead of ASCII (and remember there isn't just one EBCDIC,
   there are hundreds of them).

 . Some proprietary character set representing a non-English language.

If you choose to ignore all but Unix and DOS file systems, you still can't
dismiss line-end conflicts -- these affect real users every day, who don't
have a clue what to do about them.

It is the business of open standards -- and a founding principal of the
ARPANET --  to put common intermediate representations on the wire.  Once
you start going down the road you're suggesting, you might as well abandon
the Internet to a single vendor.

For the purposes of cross-platform file interchange, the world has settled
on distinguishing between text and binary files.  It's not ideal but taking
this concept further (e.g. converting database formats between one product
and another) are beyond the scope of open standards.  Thus text files are
candidates for conversion to common format for transmission -- and that
means both record format AND character set -- and binary files are
transmitted without conversion.  We have decades of experience to tell
us which mode of transmission is appropriate for which file.

: >We're going in circles.  The topic here was recursive downloads.
: 
: No.  The topic was wildcards in MLSD.  There are wildcards that will 
: produce matches that list the contents of one or more directories; there 
: are wildcards that match some files and some directories.  If MLSD is to 
: match wildcards, it must be done reliably, and in a manner that can be 
: replicated by client and server.
: 
Again, I'm not suggesting that MLSD accept wildcards; rather I'm proposing
that MLSD-like commands be added that do so.

Wildcards exist.  As you pointed out, RFC959 deals with them, but in a way
that has been rejected by virtually every developer.  I agree it would be
nice to have standard wildcard syntax and semantics -- but it would also be
a large undertaking.  As matters stand, server-side wildcard interpretation
offers an "escape valve" to users who know what they are doing, that lets
them accomplish effects that could not be achieved otherwise.  In a word,
server-side wildcards can be "useful".

: >: Why a fact for MODE?
: >:
: >Why not?  If FTP has a MODE command with a selection of values, and if,
: >in a certain setting, the value makes a difference, then it would be
: >useful for the server to suggest the appropriate MODE for a given file
: >(if the server knows it) so the user doesn't receive garbage.
: 
: Because every mode can be used to transfer any file.  Hence, MODE is 
: entirely the choice of the client.
: 
Right, but the server can suggest the most appropriate one, since it might
know something the client doesn't.  Stream mode is appropriate for stream
files, block mode for structured files.

: >Years ago I, too, thought inspection was gross.  Then I tried it.  In
: >Unix and Windows, the effect of looking at (say) the first 48K of a file
: >is barely measurable (unless it's on a floppy disk or something).
: 
: Are you the same Frank that was proposing the server manage a million files 
: in a directory?
: 
I'm the same Frank that has discovered that certain ideas have value in
certain situations.  Situations can exist where the benefits of the server
offering useful information about a file outweigh the performance penalties.
It is the users of FTP who should make the choice about the tradeoffs based
on their needs at the time; it's not our business to make those decisions for
them based on assumptions like "performance has precedence over utility."

- Frank