Ftp-WG: NLST vs MLSD

On Sat, 12 Oct 2002 10:31:40 -0500, Robert wrote:
: At Thu, 10 Oct 2002 10:25:57 -0500, Frank wrote:
:   | It seems to me that the MLSD spec pushes a lot of work off on the 
:   | client,
: 
: Absolutely.  Which is where it ought to be.  As a client author I know
: that means more work for you...
:
I'm sure you know my interest in this is not avoiding work.  I've already
done most of the work, so if nothing changes the Kermit FTP client will
fit in, but how will users receive it?  My concerns are:

 1. That the spec be as clean, simple, and elegant as possible to
    promote the widest implementation; not just superficially (because
    it already is clean and elegant on the surface) but also at the
    deeper level where it must be implemented.

 2. That it not force choices upon the user that s/he is not equipped
    do deal with.  This not only frustrates users, but exerts back
    pressure on help desks.  That is, it has human consequences.

I understand there's a tradeoff between the complexity of the server and
the client, but we're talking about the interface between them.  The
interface should be usable in a straightforward, obvious way or it won't
take off and we'll still be parsing "ls -l" listings for another 30 years.

:   |  . Whether the server should interpret wildcards in the MLSD argument.
: 
: Not just whether, but also what he wildcards actually mean, otherwise your
: client is going to have to interpret the user's spec, and re-code it into
: some standard wildcard language.
: 
Wildcard syntax and semantics are, by nature, up to the software that
implements them.  I don't believe this can be legislated, given the
free-for-all nature of software development.  You also can't legislate away
30 years of practice.  The protocol specification might be cleaner and
clearer by avoiding the issue, but neither users nor developers will be
happy.  Nor can you safely design a standard wildcard language because you
can't enumerate all the known and future file-system and wildard syntaxes,
and therefore can't avoid introducing conflicts and ambiguities.  At some
point, if only as a final resort, the user must be exposed to the server
host's notation -- it can't be totally hidden without interfering with
access.

As noted previously and agreed by all, RFC959 was not clear in this area,
This resulted in widespread if not universal server-side wildcard
interpration.  The cat has been loose and breeding for decades; you can't
undo this.

New servers that support MLSD as specified (and perforce, continue to also
support NLST) will force arcane choices upon the user, as noted in my paper.
We might consider it unlikely that a user might want to get 3 files out of a
directory containing a very large number files, but that does not mean the
protocol should ignore this case.  It's same argument that was made when
designing the address space of the early networks, which was outgrown faster
than any expert foresaw (more than once).

: Personally, I believe that wildcards have no place at all in the FTP spec.
: They will cause far more problems than they'd ever solve.
:
You can't ignore them.  RFC959 did and look what happened.  The spec needs
to say something about which side handles them and, since there are
legitimate and even compelling arguments to made for each side, a choice
should be offered.  Suppose we have two commands, MLSD and MLSW that are
implemented in every server.  There is a use for each.  GUI drag-and-drop
FTP clients will choose MLSD; command-line clients (which we must not
pooh-pooh) will tend to prefer MSLW because they are used by professionals
who care about -- and need -- detailed control and automation capabilities.

: Note that you cannot simply use a cdir line, require it to come first, and
: define that as the beginning of a new directory, or you would be unable to
: distinguish the case of a directory having multiple cdir lines (which is a
: very reasonable thing to do - giving both a fully qualified path, and a
: relative path from the PWD) from an empty directory, followed by another
: one.
: 
Then "cdir" needs a stricter definition; it should do one thing or the
other.  Add another fact if it's necessary to do both (and I believe it is;
the simple relative form for GUI title bars or whatever, and the fully
qualified form for use by the client in navigating the tree).

In fact, this one of the most annoying things about FTP -- you can't find
out (programmatically) the server's current directory.  The PWD command is
useless.  CDIR should be an improvement.

Ditto for PDIR.

:   |    But what about downloads?  If the server
:   |    could employ these same algorithms, then it could include a
:   |    "text-or-binary" subfact with the "type" fact, similar to the MIME
:   |    Content-Type value.  This would be a hint to the client to use the
:   |    appropriate mode for RETR, so as to avoid garbaging the file.
:   |    (Just a hint -- of course the client can do as it wishes.)
: 
: Sounds reasonable to me, write a definition for the fact type....
:
FTP protocol requires the client to control everything, but only the
server knows about its own files.  The server needs a way to suggest the
appropriate STRU and TYPE for each file to the client.  This could be done
in various ways, but the cleanest is to add new facts:

  mode=<any-mode-value>   (S, B, C)
  stru=<any-stru-value>   (F, R, P)
  xtype=<any-type-value>  (A[ x], E, I, L x)

"xtype" distinguishes this new fact from the existing "type" fact.  If a
value contains spaces (as TYPE values can), we have to work around the
prohibition against facts containing spaces, such as replacing space by
(say) an ASCII punctuation character, e.g.:

  xtype=L:8
  xtype=A:C

Presumably we won't have to confront "quoting hell" since MODE, STRU, and
TYPE values should always be alphanumeric (but watch out; I don't see this
stated explicitly anywhere).

The wording should be to the effect that each of these facts can have any
value defined for the corresponding FTP command, thus not committing
ourselves to enumerate them, which would require this spec to change any
time a new MODE, STRU, or TYPE value was added somewhere else.  But it
should list the existing values as examples.

:   |  . For cross-platform (or even same-platform) directory-tree 
:   |    duplication, it might be desirable to have a standard notation for
:   |    file permissions, as distinct from the "perm" fact, which applies
:   |    only to the current user of the FTP connection.
: 
: Im not sure how exactly that will work, as on almost all systems, the
: permissions vary from user to user.  To describe anything more general
: than the current user, you need to be able to identify users in some way.
: And that then means some kind of general user namespace management
: problem.
: 
We've had some experience with this in Kermit.  Clearly you can't cover
everything (ACLs spring to mind), and there's not much you could reasonably
expect to do about namespace (file owner and group).  But a simple model
could accommodate most things that people care about: user and world read,
write, delete, execute, list, and append permission.  Maybe group too.  This
takes in VMS, Unix, DOS, TOPS-20 (gone but not forgotten because it
pioneered many concepts some of which still need to catch on).  In any case
this is just information provided by the server to the client in a new
fact, say XPERM.  The client can use it however it wants.

In the reverse direction it is also often desirable to set permissions of
uploaded files.  FTP protocol does not give a way to do that.  Defining a
standard notation for file permissions could open the door to this.

: Yes, "executable" is perhaps one thing that is currently missing, though
: it is not clear to me whether executable is a permission, or a file type,
: or both.
: 
It's a permission because it grants execute permission.  In TOPS-20, a
program could have execute but not read permission and you could still run
it.  But you couldn't copy it, save it, or look at its contents.

Of course conceptually it's also a file type, and perhaps it might be useful
to distinguish between (say) a shell script that has execute permission and
an executable binary program (that might or might not have execute
permission), but I'm not sure I see any need for this.

Of course some file systems overload execute permission to save bits, e.g. a
Unix directory with x permission means you can list its contents, but that's
a separate concept.

: We could perhaps do with another fact type, perhaps attributes, to
: describe things that aren't really permissions, and aren't really types
: either.
;
We've done this in Kermit, but far from perfectly.  The biggest problem
comes in defining how to describe highly-structured files: fixed-length
record files, variable-length record files (as in VMS, VOS, CMS), nonstream
files (relative, indexed), various record formats (e.g. with carriage
control).  Ultimately what we settled on was a special OPTIONAL mode of file
transfer in which the file's control block is sent (in a specially coded
segment) along with the file.  If the file transfer partner is on the same
platform, it can use it to reconstruct the file; if it's not, it can archive
the information in a way that can be used if the file is subsequently
transferred to a computer with compatible FCBs.

But in the general case, I think you've already defined most of the commonly
used attributes: date, size, character-set, etc, and FTP's existing MODE,
STRU, and TYPE commands cover most of the rest.  (Any VMS FTP implementors
read this?)

:   | But as the spec currently stands,
:   | there is no way to do that without risk.
: 
: There is no way at all to use NLST the way you do, at all, without risk.
:
It's not me, it's everybody.

: You have just decided that the risk is low enough that you have decided
: to ignore it.   As server load gets higher, I fully expect to see servers
: start looking at the spec, seeing that they're not required to handle
: wildcards, and simply stop doing that (just as many that used to do on the
: fly gzip or ungzip, no longer do, because it just costs too much for the
: server, regardless of how much it would save the network or the client).
:
You're overloading MLSD.  It is at once a new file-listing format and a
new discipline to reduce load on the server -- two unrelated concepts.  It
should be possible to get the new listing format without having a whole
new set of rules about who does what.

Perhaps what we need is a way for the server to set policy.  First of all,
we have to agree to accept the fact that NLST expands wildcards because it
does; you can't change it.

Then suppose the server supports MLSD, and announces this in its FEAT
response.  The client sends "NLST [abc]*.zip".  The server says:

  550 Too busy for NLST - try MLSD

The client can switch to its MLSD handler and try again (sending "MLSD"
and then holding "[abc]*.zip" locally for matching).

:   | Thus the hideously complicated and
:   | obscure user interface described here:
:   | 
:   |   http://www.columbia.edu/kermit/newftp.html
:   | 
:   | to allow the user to get out of trouble when the client's choices are
:   | not what the user needs.
: 
: This is all because your'e attempting to optimise things which very
: rarely ever need optimising.
:
I've been in this business long enough to know that using words like
"rarely" is tempting fate.

As for the rest, we're going around in circles; I've said my piece and I
don't care that much.  If anybody else cares about these issues, please
speak up; if not, don't.

Meanwhile, in case I still have anybody's attention, here's another item
that might be worth looking at:

RFC959's STOU specification is inadequate.  First, it does not accept an
operand, which is unreasable.  You should be able to "stou foo.bar" and have
it stored as foo.bar if there is no conflict, or foo.bar.1 or whatever if
there is, and in fact many clients and many servers do allow this, despite
the spec.  Second, and more serious, there is NO WAY for the client to find
out the name picked by the server for the file.  Why should we care?  FTP is
used increasingly for transaction processing, EDI, etc (for example at IBM
Info Exchange and Advantis).  "Atomic file movement" and feedback is
essential for transaction processing.

Summary:

 . Add MLSW, MLSR, MLSX.
 . Define a standard notation for permissions.
 . Add XTYPE, MODE, and STRU facts.
 . Add an XPERM fact for sending file permissions (as distinct from PERM).
 . Add a command (PERM?) for setting server file permissions.
 . Do something about STOU: augment its definition or define a new command.

Oops, one more thing....  The SIZE command returns a different result
depending on the prevailing TYPE.  But the SIZE fact always returns the
actual length of the file in octets, right?  I just wanted to make sure
this is intentional; in fact I prefer the latter behavior.  I discovered
that if you don't switch to TYPE I before sending a SIZE command then
you can wind up waiting an awfully long time for the answer!

- Frank

P.S. Sorry for sending a "help" command to the list -- I meant to send
it to the -request address...