Re: Debian and package-framework

So, as an accidental side effect of this discussion, I wound up writing 
a little
essay about package framework.   I don't think it directly answers a
debian-maintainers concerns but I think the perspective might be
helpful anyway. 

Regards,
-t

                     Why We Use `package-framework'

  Sometimes people are surprised that we use `package-framework'
  rather than `autoconf' to configure and build our programs.

   Package-framework and the autoconf family of tools both make
  it easier to write Makefiles and configuration scripts, but
  they solve different problems.

   The autoconf family is best for bootstrapping the first few
  hard-to-port GNU programs in a new environment such as GNU
  Make, GNU Bash, and GCC.

   `package-framework' is better for most other programs:
  programs that are not so hard to port and that can count on
  the system to provide a Posix shell and GNU Make.

   * autoconf Solves a Chicken and Egg Problem;
      `package-framework' Makes a Chicken Omellete

   Consider the problem of writing a configure script and
  makefile for GNU Make and GNU Bash.  There's a
  chicken-and-egg problem.  The makefiles can't assume that GNU
  Make is already installed -- but the native versions of make
  found on many systems differ from one another greatly.
  Similarly, the configure scripts and the rules in makefiles
  can't assume that GNU Bash is already installed, yet there
  are awkward differences among the /bin/sh implementations
  found on many systems.  For those reasons, the autoconf tools
  implement "compilers" for makefiles and shell scripts.  These
  read high level descriptions and emit truly portable shell
  scripts and makefiles.

   Those compilers are tricky to write and maintain.  Worse,
  the autoconf tools encourage users to extend these compilers
  with new features, just for their own programs, and often
  users do this incorrectly -- producing a configure script or
  Makefile that will only work on a typical GNU/Linux system
  but might not work on, say, a FreeBSD system.  This is one of
  the ways that when people use autoconf other than to bootsrap
  those first few programs, the result can wind up being *less*
  portable: exactly the opposite of the reason to use autoconf
  in the first place!

   Package-framework, on the other hand, requires a standard
  Posix shell (Bash will do) and GNU Make.  Those are pretty
  small requirements.  In exchange, package-framework works
  without the need for shell script and makefile "compilers".
  This makes package-framework smaller, simpler, and easier to
  maintain.


   * `autoconf' Encourages Feature Tests,
      `package-framework' Encourages Standard Code

  Programs like GCC, Bash, and GNU Make were among the first
  written and distributed by the GNU project.  These programs
  could not assume that the GNU C library would be available
  because, at that time, it wouldn't even be complete for
  several more years.  Even today, no portable program is save
  to assume that the GNU C library is available.

  Back then many systems had C libraries and the usual header
  files but there was a problem.  Although POSIX standards
  existed, few systems implemented those standards perfectly.
  Some functions had different names on different systems.
  Some header files were in different places.  Some header
  files and functions were missing entirely.  Sometimes
  everything was in the right place, but a function was buggy,
  or had the wrong type.  All of these problems still happen
  today but to a much reduced extent: the POSIX standards are
  more widely and more accurately implemented.

  The variability between systems made it hard to write the
  first core GNU programs in a way that would compile in many
  different environments.  A simple fix for one environment
  could cause the build or the program to fail in another
  environment.  Generally speaking, people solve this kind of
  problem four different ways:

     1. conditional code based on system name
     2. conditional code based on parameters from the user
     3. conditional code based on feature tests -- probe the
        system during configuration to see what's there
     4. conservative coding and compatibility libraries

  The GNU tools use a mix of all of those but autoconf places a
  heavy emphasis on #3: feature tests.  (GCC, of course, must
  also use #1, especially learn about the CPU type.)

  Feature tests are generally better than making guesses based
  on the system name or requiring parameters from the user.
  The resulting pre-processor conditionals tend to be simpler.
  The result tends to be more robust and more easily ported.

  #4 is almost always the best choice *when the option is
  available*.  For the early GNU programs, the option was often
  not available.  There was no one portable way to do proper
  signal handling and subprocess management, for example, and
  no way to really solve that by writing a compatibility
  library.  So autoconf placed a heavy emphasis on feature
  tests and, to this day, encourages users to solve portability
  problems that way.

  `package-framework' is lucky to exist in a *slightly* simpler
  world.  It is much easier to write portable code, even for
  many low-level system operations, without any pre-processor
  conditionals at all.  So, while `package-framework' does have
  minimal support for feature tests, the emphasis is on
  conservative coding and compatibility libraries.

  * `autoconf' is Too Complicated
    `package-framework' is Too Simple

  The `autoconf' tools have been stretched far beyond their
  original mission to help porting the first few GNU tools.
  These days, they (it seems to us) are used for the vast
  majority of C and C++ programs in the free software world.
  `autoconf' has grown to be quite a bit larger than it
  started.  For some uses, it now depends on `perl'.  It has
  not gotten easier to maintain -- if anything, harder.  Many
  of us have experienced the frustration of `autoconf'-using
  programs that won't configure or compile correctly other than
  on a GNU/Linux system.  Many of us have experienced the
  frustration of having to have around multiple, incompatible
  versions of `autoconf' because some programs require one
  version while others require a different version.

   `package-framework', more or less, does one simple thing
  well.  It is already very simple and, even so: there is more
  code than needs to be there for the current functionality.
  If it changes in the future, is should become even simpler.

  But `package-framework' is in some ways a little bit *too*
  simple.  Right now, it offers no help at all building or
  linking against dynamically loaded libraries -- a major
  deficiency.  Additionally, it has no support for finding a
  required version of an installed library in the case where
  there may be more than one version installed.  These
  weaknesses are why, for example, a particular version of
  `libneon' is bundled with GNU Arch and why that library is
  statically linked to Arch.

  * A Possible Future

  `package-framework' generally serves us well and we think
  that its basic design is sound.  We believe that `autoconf'
  is over-stretched as it is popularly used and that it
  encourages people to use feature probes where more
  conservative coding practices would be a better solution.  We
  emphatically recognize that `autoconf' still does, and may
  "forever" play an important role in bootstrapping the most
  basic GNU system tools.

  The `package-framework' implementation today is a little bit
  messy.  It would be an improvement to carefully review the
  scripts and remove "dead code" and some minor "features" that
  serve no useful purpose.  It would be an improvement to
  carefully review the Makefile library and make similar
  changes.  The result would be a smaller, sleaker
  implementation of the current functionality.

  Next: `package-framework' needs support for dependencies on
  versioned libraries and for dynamic library loading.
  Versioned library support is a small matter of writing some
  new shell code -- there is no great mystery to it.  Dynamic
  loading is tricky because, indeed, this is an area where
  portability is hard to achieve.  It is not clear that
  adopting `libtool' directly is the right answer: `libtool'
  needs to be studied from a `package-framework' perspective
  and either adopted or reverse engineered and replaced.

  Next: There is a need for improved standards for packaging
  source distributions of software packages.  On all GNU
  systems, there should be a standard way to determine where
  to install programs and to tell what other programs have
  already been installed.   Configure/build tools are the
  right place to implement such standards.

  Finally, it is difficult to imagine an easy way to get many
  projects to switch to `package-framework' or to adopt new
  standards for source packages.  People have an awful lot
  invested in the tools they already use.  Vendors display only
  weak interest in that kind of standard.  We have many legacy
  packages, already very hard to configure and build, for which
  the effort of switching over would be monumental.
  Nevertheless, these are problems of the sort that can be
  meaningfully and usefully addressed in "one small step at a
  time" mode.   What will give people incentive to make the
  switch?  To have very good tools that make the switch easier
  to stomach, and to be offering very good benefits that result
  from switching.   Step 1: work on the tools, keeping the need
  for benefits in mind.




Sylvain Beucler wrote:
> On Mon, May 08, 2006 at 08:01:40PM +0200, Sylvain Beucler wrote:
>   
>> I have a few questions with regard to packaging / package-framework,
>> could you help us?
>>     
>
> I think I can answer some of them now, feel free to correct me :)
>
> - documentation: read comments in
>   src/build-tools/scripts/configure-dirs
>
> - cross-compilation: this would require passing --host/--build
>   ../configure options down to PLUGIN/AUTOCONF-based components, and
>   introducing a caching or best-guess mecanism in hackerlab's
>   compile-time tests afaics - so I can forget about it.
>
>
> - using externals libs instead of PLUGIN ones (for libneon): that one
> I still wonder about :) Is it supported by package-framework somehow,
> or will I have to trick the system? (eg. via some hand-made Makefile
> rules that would set gcc -I and ld -L appropriately?)
>
>
> Thanks!
>
>