Re: ASDF, Windows, symlinks

I'd like to start with a quote from BSD's symlink(7) man page
(http://www.stat.ucla.edu/cgi-bin/man-cgi?symlink+7), which I think is
the definitive statement on symlinks as it is from the group that
actually created them:

  Because a symbolic link and its referenced object coexist in the
  file system name space, confusion can arise in distinguishing
  between the link itself and the referenced object.  Historically,
  commands and system calls have adopted their own link following
  conventions in a somewhat adhoc fashion.  Rules for more a uniform
  approach, as they are implemented in this system, are outlined here.
  It is important that local applications conform to these rules, too,
  so that the user interface can be as consistent as possible.

That sets the stage fairly well for what I'm about to say: ASDF's
handling of symlinks is bad design and goes against the rules that
have been in place for more than 20 years.  They are good rules and
there are good reasons for them.

First, what is the behavior of ASDF that is in question?  This is from
revision 1.115 of asdf.lisp from cclan.cvs.sourceforge.net:

  981 (defmacro defsystem (name &body options)
  982   (destructuring-bind (&key (pathname nil pathname-arg-p) (class 'system)
  983                             &allow-other-keys)
  984       options
  985     (let ((component-options (remove-keyword :class options)))
  986       `(progn
  987          ;; system must be registered before we parse the body, otherwise
  988          ;; we recur when trying to find an existing system of the same name
  989          ;; to reuse options (e.g. pathname) from
  990          (let ((s (system-registered-p ',name)))
  991            (cond ((and s (eq (type-of (cdr s)) ',class))
  992                   (setf (car s) (get-universal-time)))
  993                  (s ...)
  998                  (t
  999                   (register-system (quote ,name)
 1000                                    (make-instance ',class :name ',name)))))
 1001          (parse-component-form nil (apply
 1002                                     #'list
 1003                                     :module (coerce-name ',name)
 1004                                     :pathname
 1005                                     ;; to avoid a note about unreachable code
 1006                                     ,(if pathname-arg-p
 1007                                          pathname
 1008                                          `(or (when *load-truename*
 1009                                                 (pathname-sans-name+type
 1010                                                  (resolve-symlinks
 1011                                                   *load-truename*)))
 1012                                               *default-pathname-defaults*))
 1013                                     ',component-options))))))

Line 1010 contains a call to resolve-symlinks, which causes pathname
for a system to be the resolved symlink of the file in which the
defsystem appears (if :pathname is not specified in the defsystem
options).  Specifically, if /build/module/foo.cl is a symlink to
/src/module/foo.cl and foo.cl uses asdf:defsystem, then as far as ASDF
is concerned, the pathname of the module is /src/module/ and not
/build/module/.  This means the compiled versions of source files for
the system will go into /src/module/ and not /build/module/.

Second, what rule does ASDF break?  In symlink(7) lingo, ASDF either
falls into category #1 or #2, depending on how you view it.  Clearly
ASDF does not fall into category #3, as ASDF is does not traverse a
file tree.  If you disagree, then think about make(1), which is the
most ASDF-like tool in UNIX, and it is not in category #3.

Third, why is breaking ASDF the rules a bad thing?

 1. It means ASDF doesn't behave in a way that would give the obvious
 behavior.  These rules in question are there because they allow
 simple prediction of behavior, and construction of tools that rely on
 this behavior.  ASDF toss this long assumed behavior out the window.

 2. It is impossible to get the "correct" behavior out of ASDF.  That
 is, the decision made by the ASDF authors means certain things are
 impossible with ASDF.

 3. Hooks could be added to ASDF that would allow backward compatible
 behavior.  That would allow both the old behavior and behavior that
 is currently excluded to be possible.

Now, an example.

Let's say I want to have one copy of my source code on the network and
build from that.  I build in parallel on many machines at the same
time, and they all symlink to the single set of sources (directories
are no linked to, only source files).  The results of the builds are
local, only the source code lives on the remote filesystem.  This type
of situation is common and has been employed for more than 20 years.

Because of the way ASDF operates on the resolved name, the above is
not possible.  That is, the fasl files for all parallel builds end up
in the same place using the same names.  (Yes, one solution to this is
to use different file extensions for each architecture, but that is a
workaround to a broken design.)

Before people start saying that this example is Franz specific:

* http://bugs.mysql.com/bug.php?id=18888

  "Our software-distribution system symlinks all source-files from
  another location (read-only filesystem) before build."

* http://cvsman.com/cvs-1.12.12/cvs_113.php

  "Another approach is to set up a structure which allows
  developers to have their own copies of some files, and for other
  files to access source files from a central location. Many people
  have come up with some such a system using features such as the
  symbolic link feature found in many operating systems, or ..."

* http://hackage.haskell.org/trac/ghc/wiki/Building/Using

  "So for every source tree we have zero or more build trees. Each
  build tree is initially an exact copy of the source tree, except
  that each file is a symbolic link to the source file, rather than
  being a copy of the source file."

Regarding the message to which I'm actually replying, the idea of
using Windows shortcuts is such a bad idea it boggles the mind.
Nikodemus summarized well the issues in his email, but I'll repeat:
Windows shortcuts are an explorer (GUI) hack, they are not built into
the filesystem (NTFS/FAT32).

The right thing to do is to change the design of ASDF to not manually
indirect through symlinks.  That would solve the problem with Windows
and make ASDF have predictable and reasonable behavior.

I'll end with a question.  Since something drove the authors of ASDF
to make the design choice they did, can someone shed some light on
what it was?  As is often the case with bad design choices, there is
sometimes a better solution.  Perhaps the original problem can be
framed here so that we can all help think of a better solution than
what we have.

Kevin
Franz Inc.

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/