Re: Draft manifest

On Mon, May 06, 2002 at 02:51:39PM +0100, Mark Murray wrote:
> Said Jarkko Hietaniemi <[email protected]>:

> >                         Well, modulo Perl releases, of course, so it's
> > more like saying "Hi, I'm one of the official Perls, you can use
> > me."-- and official Perls do include the respective modules.
> 
> My reading is that official perls _happen_to_ include the respective
> modules. It is with some dismay that I discover that this is not
> the case, but I am heartened by another direction. That direction
> is the one suggesting that perl should be split into "base perl"
> + "standard libraries", with the ability to install only "base"
> into an OS with the libraries as an option.
> 
> Having the libraries as part of the base language for Perl makes
> about as much sens to me as making glibc a part of C or Microsoft's
> Foundation Classes a part of C++. You may _need_ these things, but
> they are separate entities.

[I hope I have the correct terms here:]

A better analogy is how everyone expects an ANSI conformant C implementation
to have the ANSI library with all the headers.
*However* we're all used to hosted C implementations, and forget they're not
the only sort.  Which is why it comes as a bit of a shock when WinCE turns
out not to have errno.h.  Which it is allowed to do, as Microsoft in their
wisdom decided to only provide a freestanding implementation.

On Mon, May 06, 2002 at 05:03:36PM +0300, Jarkko Hietaniemi wrote:
> On Mon, May 06, 2002 at 02:51:39PM +0100, Mark Murray wrote:

> > I never read it that way. I've never seen anything saying that use
> > of the core language means you have to swallow the libraries as well.
> 
> That's the only sensible interpretation (again, from the viewpoint of
> a Perl user, the viewpoint of a "kernel-builder" might be different).
> (Any other interpretation would be painful to document.) 
> 
> If one needs to know that in platform FOO the standard modules BAR and
> ZOG are not available, Perl has just lost one of its major strengths,
> portability.  The idea is that once you know what Perl release you
> have, you know what toolkit you have.
> 
> But yes, maybe it's the to officially considering breaking that
> dependency.

A good example of the problem is the NMS project (the lot I seem to be
promoting in my .sig despite having not actually read any of the code, let
alone used it.)
Offhand I can't say whether the minimum release is 5.004 or 5.004_04.

However, the importance of this is not just avoiding later perl syntax.
The scripts attempt to promote good programming practice including code
reuse by using the modules present in the core perl release.  The authors
know what modules are present at by that specific version, and so use them
without any expectation that they may not be there.

Providing something billed as "perl 5.004_04" without it being clear that
as is it's less than the perl distribution you'd get from CPAN is going to
give people an unpleasant surprise.  And it's quite likely that they're going
to be confused and think that it's buggy (for some definition of somebody
else's fault).  And 5.6.1 just like 5.004_04.
[And it's a reasonable guess that nms *is* expecting CGI.pm, seeing as it
sets out to better Matt's Script Archive :-)]

On Mon, May 06, 2002 at 03:42:05PM +0100, Mark Murray wrote:
> > On Mon, May 06, 2002 at 11:31:26AM +0100, Mark Murray wrote:

> > The important bit being that it would be really useful if we can make it
> > so the user doesn't need to download a new perl binary to do this.
> > Just add dynamic linked extension to the base system's perl.
> 
> How? "Adding dynamic linked extension" is either rebuilding the perl
> binary or loading another one. (Binary editing would be laughed out
> of town!)

Aargh. Not clear. I meant having the otherwise minimal perl ship with
DynaLoader.  ie ship a complete perl binary, just don't ship many .pm or
any .so files. Have them all available as 1 or more packages.

> > It would be useful to be able to trim it so that a self-hosting perl
> > distribution (ie one capable of getting and installing more modules without
> > outside help) could be as small as possible.
> 
> You could go for a dual system. Self-hosted-perl-only or
> self-hosted-but-using-OS-apps.

That makes sense. I think there's still a logical progression:

0: micro - just enough to be useful. Almost everything in perlfunc works,
   dbmopen doesn't work. Relies on some other packaging system to extend it
   maybe with DynaLoader?


1: mini
   As above plus
   Has enough to build extensions, but does rely on OS supplied tools to get
   the source. (eg command line ftp client, lynx/w3m/links, tar, gzip, unzip)

2: mini self hosting
   As above plus
   Doesn't need any OS tools to get things.
   [not sure how decompression works. Reading tars in perl ought to be easy,
    it's the gunzip bit]

....: everything else


> > > >    I think it must have Fcntl, File::Glob and SDBM_File in ext (see below)
> > > 
> > > Pretty much most of what is there now. Notable exception - CGI.pm.
> > 
> > Well, I meant of 5.8's ext:
> > 
> > B           Digest      Filter     MIME       SDBM_File  Time     threads
> > ByteLoader  DynaLoader  GDBM_File  NDBM_File  Safe       Unicode  util
> > Cwd         Encode      I18N       ODBM_File  Socket     XS
> > DB_File     Errno       IO         Opcode     Storable   attrs
> > Data        Fcntl       IPC        POSIX      Sys        attrs.t
> > Devel       File        List       PerlIO     Thread     re
> > 
> > the only 3 dynamic extensions would be Fcntl, File::Glob and SDBM_File
> 
> The rest being "optional"? That I can swallow. No CGI? Hmmm. It was CGI
> that sparked off this furore :-).

I think the furore was that by just removing CGI you broke our assumption
(or contract) - that if a perl says it's version N, then it comes with all
the library that version N shipped with.

Debian managed to break the contract at the change from the previous to the
current perl maintainer when (due to some accident it seemed) they removed
support for parallel perl versions.  Previously they had 5.004, 5.005 and 5.6
all packaged.  They screwed up (presumably in the dependencies) the
transition, because apt would happily upgrade your system, leaving
/usr/bin/perl5.005, but removed all the libraries.
Any scripts users wrote that used 5.005 now died, unable to find libraries.
[The correct thing to do would have been to have enough sane dependencies to
stop apt upgrading away 5.005.  Rather than doing it all silently so that
the first my boss knew about it was somewhere on a train from Newcastle to
London when he turned his laptop on]

This is sort of an aside, but the moral is that many people expect "perl"
to mean the whole distribution.

Why I say everything else is optional is this:

If you just remove CGI then it's not clear that you're changing that
assumption about what they've got.

*However*, if you remove everything you don't need from the FreeBSD 5.0 core,
then it's obvious to anyone clueful that you've done something.
Also, once you've broken the assumption by removing CGI, you've done the
important bit. So why not do the dieting job properly? :-)
For the FreeBSD core remove every module from lib and ext that you don't
need to rebuild the core. Put them all in a perl package. And have the release
notes/upgrade notes for 5.0 series FreeBSD explain the situation. I'm
envisaging something like

"The FreeBSD 5.0 core doesn't contain a full perl distribution. Although
 /usr/bin/perl is present, most of the libraries are not present.
 The supplied bare bones configuration is intended solely for FreeBSD's make
 world, not for running general user perl programs, such as for administration
 or production tasks.
 If you or your users want to use perl scripts you are strongly recommended to
 install the perl package, which provides a full perl install."

where "perl" may not be the best name for the perl package, and "make world"
might also be a stupid thing to call it.

This way the FreeBSD core gets a lot smaller, and there should be no
confusion about what's going on.

As I understand both sides positions as this

FreeBSD: We'd like to have the perl interpreter available to remake the
         FreeBSD userspace and kernel, but the full libraries that come with
         it are far too big a burden for the core.
         We'd like to provide the rest of perl as a package, so that those
         want to rebuild the kernel but don't want to run perl themselves
         aren't forced to have perl present.
p5p:     When we say perl version X, we mean both the interpreter and all
         the libraries that were released in the source package for X.
         Many scripts make assumptions about which modules are present based
         on the version of the interpreter they are running under, and p5p
         considers this supported behaviour.  Hence we don't want scripts
         (or users) to encounter something they think is perl version X
         which doesn't have all the libraries, and doesn't say that it doesn't
         fulfil this expectation of all the libraries

As I see it (and I'm not all of p5p, nor the pumpking, nor Larry Wall)
providing /usr/bin/perl and some subset of libraries is fine, providing it's
clear to all using it that you've made changes from the official perl
distribution, and what they are.
[well, "we removed nearly all the modules" is clear to me.]

> > I'm confused. DynaLoader has to be staticly linked. So if the system core
> > perl binary doesn't have DynaLoader linked in it would have to be replaced
> > with a perl that did have DynaLoader.
> 
> Yes. This means that the base/minimal perl should ideally have Dynaloader.

because then the system core perl binary is /usr/bin/perl, and stays the
same binary when the user decides they want the perl-borg package, and stays
the same binary when they decide they're uninstalling it :-)

> Base perl (including dynaloader) smells ideal to me.

Not sure how glob() works though. perl-borg would want to have /usr/bin/perl
do glob() with File/Glob.so
minimal perl wouldn't want File/Glob.so, but (I presume) would want glob()
to work.

> > > Why would Fcntl be a major requirement? Where would a bootstrapping
> > > process need it for?
> > 
> > Anything that wanted to do fcntl or ioctl with the proper symbolic constants
> > would need Fcntl.
> 
> And a toolkit to build it. Careful, here! A FreeBSD 'make world'
> would not want perl to dig around in /usr/include; the files to be
> grovelled through have not net been installed.

Building Fcntl doesn't need any digging in /usr/include
Building the module from Fcntl.xs doesn't do any (perl) scanning or parsing
of any include files. Someone once did so with h2xs to make Fcntl.xs, but
that's a long time ago. [Whatever scanning gets done is by the C compiler
as it compiles Fcntl.c]
But you do need a working perl to convert Fcntl.xs to Fcntl.c

However, you don't need the latest perl to do this. What's wrong with
building this extension in /usr/src/gnu/usr.bin/perl/library/Fcntl
with /usr/bin/perl during make world?

Nicholas Clark
-- 
Even better than the real thing:	http://nms-cgi.sourceforge.net/