Ftp-WG: NLST vs MLSD

At Wed, 9 Oct 2002 23:17:50 -0500, Alun wrote:
> Does each MLSD output for a wildcard spec even produce a 'cdir' type?
>
No.

> After all, the wild card isn't going to match a directory, usually, yet
> 7.3.1 of the MLST draft notes that an MLSD response MAY include one or
> more "cdir" type entries, probably because the authors anticipated that
> MLSD's argument would always be a directory that would be listed in that
> 'cdir' entry.  "MLSD *.txt" could produce a line that's of 'cdir' type,
> and whose name definitely doesn't match "*.txt".
> 
> In the instance that an FTP server allows "MLSD */*.txt", it may even be 
> useful to define that a 'cdir' entry should be listed prior to the files 
> that match the wildcard spec in its directory.
>
I suggested this too.  It seems to me that the MLSD spec pushes a lot of
work off on the client, and something doesn't feel right about that.  One
server, many clients -- why should all the clients have to deal with the
fact that the server doesn't order its list in a useful way?

> So that if "*/*.txt" 
> matches "a/1.txt", "a/2.txt" and "b/3.txt", you'd have a 'cdir' line that 
> specified where 'a' is, followed by listings for "1.txt" and "2.txt", then 
> a 'cdir' line for 'b', and a listing for "3.txt".
> 
> This goes against the note in 7.3.1 that the 'cdir' entry can occur 
> anywhere in the MLSD output, because it would require that the 'cdir' line 
> precede all matched elements in that directory.  Would it be worth 
> specifying that all 'cdir' output lines must precede other lines relevant 
> to that directory?
> 
I have been advocating that.  If "cdir" can't be taken as a heading, what's
the point of including it in an MLSD listing if the client can't tell which
entries it applies to?  (The answer is evidently something like: "it can be
used as a heading, in the sense that a GUI might put the cdir name in the
title bar, but it isn't a heading in the sense that it comes at the head of
the output")

> Here's another thought - what should MLSD do if given a wildcard that 
> expands into several names, some of which are directories, others of which 
> are files?  Should it list the details of the directories, or the contents 
> of those directories?
> 
Since there are many possibilities, I think we might want to define a
family of MLSx commands; one to do each thing.  Or perhaps more rationally,
to allow MLSD to be accompanied by options ("switches") because you quickly
get into combinatorial problems if you have to define a separate command
for each combination of options.  Just off the top of my head, I think the
options we need include:

 . Whether the server should interpret wildcards in the MLSD argument.
 . Whether the server should return a recursive listing.

Well, if there are only two options, maybe it's not so bad:

         Interpret 
         Wildcards   Recurse
  MLSD      No         No 
  MLSR      No         Yes
  MLSW      Yes        No
  MLSX      Yes        Yes

So far I have dealt with recursion depth-first, by CWD'ing to any "dir" I
run across, and CDUP'ing when done (recursively).  If I had to deal with
cdir's to recurse, I'm not sure how I'd do it, e.g. to do a breadth-first
traversal (as Robert once suggested, to avoid the too-many-temp-files-open
problem).  After all, what does cdir tell you?  Not much -- usually it's
just "." or "/".  Even if I know that foo is bar's parent, how do I get
back back to foo from some arbitrary place without knowing foo's full path?

In the general case, where a directory is too deep for the straightforward
walk, I think the best approach is having MLSR or MLSX return filenames in
TVFS notation, relative to the server's current directory at the time it
received the MLSR or MLSX command.  Then we need only one temp file and we
don't have to clutter the connection with CWD or CDUP commands or do any
bookkeeping.  Thus perhaps MLSR and MLSX should be defined as returning
only filenames, only in TVFS notation.

As an aside, Kermit protocol has used TVFS-like notation for years to
convey directory structure between client and server, and it works very
nicely.  For example, a VMS server (whose path syntax bears no resemblence
to Unix's) can send a directory tree to a Windows client (Unix-like
directory structure but with different dirseps), switching automatically
between text and binary mode for each file.  The same tree can be uploaded
from Windows to another VMS computer with pretty decent results.  (Obviously
there's a lot I didn't say about preservation of VMS file properties, but
that's irrelevant here.)

But this does raise some interesting possibilities for MLSD facts:

 . Over the past few years, we have proven in the field (with Kermit
   software) that files can be categorized as text or binary for purposes
   of transfer by algorithm with near perfect accuracy.  Of course the
   algorithm varies by platform, but that's as it should be.  We use these
   same techniques in the Kermit FTP client when sending a group of files
   (MPUT).  The client can automatically switch between "ASCII" (text) and
   binary mode for each file.  But what about downloads?  If the server
   could employ these same algorithms, then it could include a
   "text-or-binary" subfact with the "type" fact, similar to the MIME
   Content-Type value.  This would be a hint to the client to use the
   appropriate mode for RETR, so as to avoid garbaging the file.
   (Just a hint -- of course the client can do as it wishes.)

 . For cross-platform (or even same-platform) directory-tree duplication,
   it might be desirable to have a standard notation for file permissions,
   as distinct from the "perm" fact, which applies only to the current
   user of the FTP connection.  Kermit protocol has this and it works out
   nicely -- for example, we set the execute bit automatically when
   receiving executable files, and we don't open up private files to world
   access unintentionally just by transferring them to another computer.

> >In any case, my concern is that the new spec conflicts with existing
> >practice.  For decades it has been possible for FTP clients to execute
> >commands like:
> >
> >   mget pics/mae-west-*.jpg
> >   ...(etc)
> >   mget pics/*/mae-west-*.jpg
> >
> >and get at least a semi-expected result.  This is about to change.
> 
> Hardly - NLST isn't going to go away.  It is still expected that NLST will 
> be used for MGET.  What you're hoping to do is to _use_ MLSD to get facts 
> about those files before transferring them, so that you don't have to do a 
> SIZE, MDTM, etc, for each file that you transfer (you could just do an 
> MLST, but I can see why you'd like an MLSD with wildcards).
> 
My point was really this: the use of NLST versus MLSD should be transparent
to the user.  Thus the client software would pick the best method in each
situation, according to the tradeoffs.  But as the spec currently stands,
there is no way to do that without risk.  Thus the hideously complicated and
obscure user interface described here:

  http://www.columbia.edu/kermit/newftp.html

to allow the user to get out of trouble when the client's choices are not
what the user needs.

> >That's a topic for another day.  For now, there is no standard wildcard
> >format, not even within one platform (with a few exceptions).  In Unix,
> >for example, every program that expands wildcards does it its own way,
> >either with its own custom code, or by calling some nonstandard library
> >or other.
> 
> Actually, to judge from the recent problems with "../*/../*" overloads, 
> they mostly use glob() from libc.
> 
Glob is not universal, nor is ftw, nftw, nor fts, nor, for that matter,
Unix.  Internet standards are platform-independent and must not assume
anything about what platforms are in play at the client or server end.

By the way, since there is so much confusion over what NLST should do, I
think there should be statement of clarification, to the effect that "in
view of widespread, if not universal, current practice..." NLST is
expected to expand wilcards and return a list of the files that match.

And that wildcard syntax and semantics depend on the software (client,
server, libraries, OS, whatever) -- there is no standard, no universal
syntax.  Unless we want to tackle that one too.  Personally, I think it's
better to "let the 100 flowers bloom" in this area.  Or at least let
sleeping dogs lie.

- Frank