Re: Perl 'Expert' Quiz-of-the-Week #24 (Module dependency evaluation)

Randy W. Sims wrote:
> 
> If you've ever written any modules then you are probably familiar with
> the concept of requirements or prerequisites.
...
> This weeks quiz is to write a module (Prereq::Expr) that can take a 
> specification of the type described above, and evaluate it to determine 
> if the requirements are satisfied.

Well, I don't have a solution to contribute, but I will say I'm happy
to see that others are thinking about this problem. If you'll indulge,
I'd like to share with you why I think this is both (a) extremely
important, and (b) only half of the problem.

One of my chief complaints with most software packages -- perl among
them -- is that their installation & usage paradigm doesn't account
for the kind of stability you need in an enterprise environment. By
default, perl is installed in /usr/bin/perl, and if you need a module
you use CPAN.pm to install it in /usr/lib/perl5 or some such. When
following prerequisites, CPAN defaults to installing the latest
available version of each module, regardless of what's requested. If
you have an old version of some module, CPAN will happily install a
newer version right over it. Even perl itself can be upgraded using
CPAN. But what if you need to install multiple versions of perl? Or
multiple versions of all of your modules?

At this point you're probably saying WHOA -- why would you need more
than one version of perl installed? And modules -- who would ever need
more than one version of a module (DBI, for example) installed at the
same time?

On your personal Linux machine, it's probably not a concern. When you
find you need a new version of something, you just install it over the
old one and expect that everything will continue to work. Even in a
small business with several unix servers, it's probably manageable --
you upgrade each server separately, making sure the scripts that run
on that server don't break.

But in an enterprise environment, with thousands of servers (spanning
multiple architectures) and hundreds of thousands of perl scripts, the
picture is a little different. Let's say I write a script today, when
perl 5.8.4 is installed as /usr/bin/perl and DBI 1.42 is installed in
/usr/lib/perl5:

  #!/usr/bin/perl

  use DBI;

In that script, I'm going to make several assumptions about the
features (and bugs) of both perl and DBI. If someone comes along and
upgrades either one, it's possible those assumptions will change and
my script will no longer work. (Moving from perl 5.6 to perl 5.8, for
example, is in many ways a PORTING exercise. And modules that work
with both still need to be recompiled when changing perl versions.)

In the casual environment, this might not be a problem -- you upgrade
something, notice that one of your scripts breaks, and go fix it. But
in a mission-critical environment -- say, a nuclear reactor, or yes, a
Wall Street trading system -- this is not acceptable. The assumptions
I make when I write a script must hold true FOREVER, or until I
explicitly change them. My script must ALWAYS run with perl 5.8.4 and
DBI 1.42, or the results will be unpredictable.

Further, maintaining that many servers brings some unique challenges.
Obviously you need your software installed on a networked file system
with no-touch installation. But you also need an unusual kind of
cross-platform consistency -- given a mix of Linux, Solaris, IRIX,
etc. servers, each script should behave exactly the same way. I should
be able to log in to any one of the thousands of unix servers in my
enterprise, run a perl script with its full path, and have it work
exactly the same way it did for the person who wrote it, even if that
was five years ago on an ancient release of some operating system that
we've since stopped using.

The only way to do THAT is to have versioned installations of
EVERYTHING, and use nothing that any OS vendor gives you, and nothing
installed locally on any machine. With perl this can be quite tricky.
Take DBD::Sybase for example: each version must be built for each of:

  - several supported versions of perl
  - several supported operating systems
  - one or more specific versions of the underlying Sybase OpenClient
    library

Some modules with more external library dependencies have even more
dimensions. The problems really start to appear when you try to
combine different modules with conflicting sets of dependencies. For
example, let's say I have two different database utility modules. One
uses DBD::Sybase 1.01, which in my environment is built against
version x of Sybase OpenClient. Another uses DBD::Sybase 1.04, built
against version y of Sybase OpenClient (to take advantage of new
OpenClient features, but that perhaps breaks some things that worked
with the old version). You can see where this gets ugly.

To rag briefly on perl, I don't think the creators or community fully
understand this problem. Of course, perl is actually a good bit better
than many other packages, in that it at least has native support for
the cohabitation of multiple arch-specific packages. But the
versioning problem adds a serious level of complexity. And when
installing a new module from CPAN, it can be a lot of work to find all
of the correct version-specific paths to 'use lib'. (In our current
environment, NO non-core modules are available without specific 'use
lib' lines, and maintaining the dependency & compatibility matrix is
quite a chore. It can also be confusing, to say the least, to
developers who don't have an in-depth understanding of how our perl
infrastructure is laid out.)

But in perl's defense, I think this was a unix problem long before it
was a perl problem. In a heterogenous environment you can't assume
ANYTHING about compatibility -- we were even bitten by a bizarre
change in /usr/bin/tr, a staple for shell scripts, between Solaris 2.5
and 2.8. And one Linux vendor in particular has taken to "helpfully"
bundling several libraries in /usr/lib, but then upgrading them
between releases without regard to compatibility, and we've therefore
had to jump through hoops to PREVENT our critical applications from
using them.

Well, sorry to rant on for so long, but I hope this has been
educational. And I hope future iterations, like perl6, will make it
easier to make sense of the version compatibility problem, in the
core, in modules, and in the dependencies between them...

--
Ron Isaacson
Morgan Stanley
ron.isaacson-/PgpppG8B+R7qynMiXIxWgC/[email protected] / (212) 762-6079

NOTICE: If received in error, please destroy and notify sender.  Sender
does not waive confidentiality or privilege, and use is prohibited.