Ftp-WG: NLST vs MLSD

    Date:        Thu, 10 Oct 2002 10:25:57 -0500
    From:        Frank da Cruz <[email protected]>
    Message-ID:  <CMM.0.90.4.1034262924.fdc@watsol>

  | At Wed, 9 Oct 2002 23:17:50 -0500, Alun wrote:
  | > Does each MLSD output for a wildcard spec even produce a 'cdir' type?
  | >
  | No.

You mean it wouldn't.   As defined, MLSD is only for directories, the 'D'
there is definitely "directory".

  | It seems to me that the MLSD spec pushes a lot of work off on the client,

Absolutely.   Which is where it ought to be.   As a client author I know
that means more work for you, but I'd much rather have you do the work,
than have server implementations doing more work all the time.   Busy servers
already have too much to do, many would like to be able to offer more, but
are already out of processing resources.   On the other hand, most of the
time, the client is just spinning, waiting for a file to transfer.

  | One
  | server, many clients -- why should all the clients have to deal with the
  | fact that the server doesn't order its list in a useful way?

One reason is because there's no definition of what is "useful".   I expect
listings to be ordered by file size, by modification time, by file "type",
by name, ...   The server has no idea what will be useful, so either it has
to be told, in which case it has to do the ordering every time - and in that
scenario, just as above, it is better for the client to do it.   Or it
simply has to decide "this is the correct order", in which case clients need
the ability to reorder anyway, to get the order that the user prefers.

  | If "cdir" can't be taken as a heading, what's
  | the point of including it in an MLSD listing if the client can't tell which
  | entries it applies to?

It applies to all of the entries, by definition.   The output is a listing
of the contents of one directory, cdir gives the name of that directory.

  | (The answer is evidently something like: "it can be
  | used as a heading, in the sense that a GUI might put the cdir name in the
  | title bar, but it isn't a heading in the sense that it comes at the head of
  | the output")

No, it isn't a heading at all.   It (when TVFS is supported anyway) can be
used to construct a path to a file in the directory, from the current working
directory of the (instance of) the server.

That is, it supplies information useful for naming the files listed.

Sure, some GUI clients might decide that they can use it in a title bar, or
something similar, but that's a by product, not the purpose.

cdir not coming first is just because that's another constraint upon the
server, which in some odd cisumstance, might mean that the server would need
to generate the listing, beore transmitting any of it.   Servers are busy.
The less restrictions we impose upon them, the better.

  |  . Whether the server should interpret wildcards in the MLSD argument.

Not just whether, but also what he wildcards actually mean, otherwise your
client is going to have to interpret the user's spec, and re-code it into
some standard wildcard language.

Personally, I believe that wildcards have no place at all in the FTP spec.
They will cause far more problems than they'd ever solve.   All we need is
for clients to start acting more reaosnably.

  |  . Whether the server should return a recursive listing.

If this was to exist, it would certainly want to be a new command, rather
than a switch, as the output format is necessarily going to need to differ
(so it is possible to distinguish one directory from the next).

Note that you cannot simply use a cdir line, require it to come first, and
define that as the beginning of a new directory, or you would be unable to
distinguish the case of a directory having multiple cdir lines (which is a
very reasonable thing to do - giving both a fully qualified path, and a 
relative path from the PWD) from an empty directory, followed by another one.

  | After all, what does cdir tell you?  Not much -- usually it's
  | just "." or "/".

If you CWD first, and just "MLSD" all the time, then yes, "." is quite
often going to be the result.   But if you give an arg to MLSD, so you're
referring to some other directory, then "." would simply be wrong (well,
assuming traditional unix type semantics anyway, there's nothing in the MLSx
spec that requires this, "." could easily be the name of some sub-directory
of the current directory).

  | Even if I know that foo is bar's parent, how do I get
  | back back to foo from some arbitrary place without knowing foo's full path?

There's no guaranteed way of doing that, other than establishing a new
connection, in FTP.   CDUP is optional.   MLSx type pdir is optional.
FTP servers aren't required to provide you with a way to get back out of
a directory they have let you into.   Once you're there, you can be confined
there.

When TVFS exists (which is probably going to be most of the time), that's
why it makes more sense to construct paths, and use them, rather than CWD
around all the time.

This is true regardless of the MLSx spec.

  | In the general case, where a directory is too deep for the straightforward
  | walk, I think the best approach is having MLSR or MLSX return filenames in
  | TVFS notation, relative to the server's current directory at the time it
  | received the MLSR or MLSX command.

That would be an option where TVFS exists.   But what do we do with the
servers that can't support TVFS?

  | Then we need only one temp file

You only need one temp file anyway.   You can easily seek around in that
as you need to.   Sure, seek pointers (and an accompanying size) are a little
more overhead for the client, than a file descriptor, but they're much less
overhead for the system that is running the client.

  |    But what about downloads?  If the server
  |    could employ these same algorithms, then it could include a
  |    "text-or-binary" subfact with the "type" fact, similar to the MIME
  |    Content-Type value.  This would be a hint to the client to use the
  |    appropriate mode for RETR, so as to avoid garbaging the file.
  |    (Just a hint -- of course the client can do as it wishes.)

Sounds reasonable to me, write a definition for the fact type....  Once
this draft gets published (eventually) we will have a fairly standardised
method of adding new facts.   Certainly it was never imagined that the
set that is in the doc will cover everything (even if we are still hanging
onto the completely useless, as far as I can tell, create-time ...)

  |  . For cross-platform (or even same-platform) directory-tree duplication,
  |    it might be desirable to have a standard notation for file permissions,
  |    as distinct from the "perm" fact, which applies only to the current
  |    user of the FTP connection.

Im not sure how exactly that will work, as on almost all systems, the
permissions vary from user to user.   To describe anything more general
than the current user, you need to be able to identify users in some way.
And that then means some kind of general user namespace management problem.

  |    Kermit protocol has this and it works out
  |    nicely -- for example, we set the execute bit automatically when

Yes, "executable" is perhaps one thing that is currently missing, though
it is not clear to me whether executable is a permission, or a file type,
or both.

We could perhaps do with another fact type, perhaps attributes, to describe
things that aren't really permissions, and aren't really types either.  The
binary/ascii file transfer mode hint, whether the file is an executable one
(which is not the same as "the current user should be allowed to execute it")
etc, could perhaps fit there.

  | My point was really this: the use of NLST versus MLSD should be transparent
  | to the user.

To the user, yes.   Not to the user's software though.

  | Thus the client software would pick the best method in each
  | situation, according to the tradeoffs.

Yes.

  | But as the spec currently stands,
  | there is no way to do that without risk.

There is no way at all to use NLST the way you do, at all, without risk.
You have just decided that the risk is low enough that you have decided
to ignore it.   As server load gets higher, I fully expect to see servers
start looking at the spec, seeing that they're not required to handle
wildcards, and simply stop doing that (just as many that used to do on the
fly gzip or ungzip, no longer do, because it just costs too much for the
server, regardless of how much it would save the network or the client).

  | Thus the hideously complicated and
  | obscure user interface described here:
  | 
  |   http://www.columbia.edu/kermit/newftp.html
  | 
  | to allow the user to get out of trouble when the client's choices are not
  | what the user needs.

This is all because your'e attempting to optimise things which very
rarely ever need optimising.   Just fetching the directory listing, and
doing your own wildcard processing would work fine in just about every
case.   It is likely to make your code bigger, and more complex, but
someone has to do all the work, and the client is the place to do it,
not the server.

  | Glob is not universal, nor is ftw, nftw, nor fts, nor, for that matter,
  | Unix.  Internet standards are platform-independent and must not assume
  | anything about what platforms are in play at the client or server end.

Absolutely.   So, when I say "mget *.txt" I should get all the files that
end in .txt on a dos or unix platform, but the file whose name is "*.txt"
on a macintosh (I think there is no magic '*' there, at least before MacOS X)
Then if I'm on a unix client and do 'mget [a-c]*' I should get all files
that start with a b or c, on a dos client I should probably get an error
as '[' means nothing to dos users (I believe), and also is illegal in file 
names.

Anything other than this means that file names given to ftp don't mean the
same as file names given to other commands on the system, and that's a very
bad UI design.

  | By the way, since there is so much confusion over what NLST should do, I
  | think there should be statement of clarification, to the effect that "in
  | view of widespread, if not universal, current practice..." NLST is
  | expected to expand wilcards and return a list of the files that match.

I would strongly object to that, and in fact, am quite likely to disable
wildcard processing in my server.

  | And that wildcard syntax and semantics depend on the software (client,
  | server, libraries, OS, whatever) -- there is no standard, no universal
  | syntax.  Unless we want to tackle that one too.  Personally, I think it's
  | better to "let the 100 flowers bloom" in this area.  Or at least let
  | sleeping dogs lie.

You mean you're proposing to require a mechanism that no-one can use safely
as no-one knows what it might mean?   That would be an "interesting" design
choice.

kre