SA-ML anyone? (oh so very long)

[Warning: another of my long rambling essays follows.]

--- XML: not just a verbose data format! ---

I've been doing a bunch of stuff with XML recently and learning a lot more
about XSLT/XPath/namespaces/etc. than I knew before. I've been wondering if
we could use XML for more than just a conveniently parsed config format.

You may have read already my murmurings on making Arusha more regular and
extensible. I was focusing before on metathings and the Python object model
side of things, but I've been thinking more recently that perhaps we should
look more closely at the configuration files themselves.

I've been inspired by XSLT (which, I've been delighted to learn Will, is
just a functional language!) to rethink Arusha and the idea of documenting/
automating sysadmin practice.

Consider for a moment, a basic GNU-style package spec:

  <package name="python--2.2.1" xml-version="1">

  <status>revealed</status>

  <prototypes>
    <prototype team="." name="GNU"/>
    <prototype team="." name="ALL"/>
  </prototypes>

  <assemble-source>
    <param name="PKG_TGZ_FILE">
      @package:ark-dirs:PKGS_SRC@/Python-2.2.1.tgz
    </param>
  </assemble-source>

  <configure>
    <param name="extra_configure_args">
      --with-pymalloc --with-cycle-gc --with-threads
    </param>
  </configure>

  <check>
    <constraint>
      <dependency type="essential" name="." on-method="compile"/>
    </constraint>
    <code lang="none"/>
  </check>

Roughly speaking, the schema for this is the Arusha schema and we manage the
"specialisation" of this particular package using prototypes and
inheritance, i.e., we know this is a GNU-style package because it inherits
the GNU-style package functionality.

But here's an idea: what if we marked this as being a GNU-style package by
using a specific GNU-style schema. Perhaps something like:

  <gnu:package name="python--2.2.1"
    xmlns:gnu="http://ark.sourceforge.net/gnu">

    <gnu:source>Python-2.2.1.tgz</gnu:source>

    <gnu:configure>
      <gnu:comment>
        We configure Python to use the new memory allocater,
        the new cyclic garbage collector, and threads.
      </gnu:comment>
      <gnu:args>
        --with-pymalloc --with-cycle-gc --with-threads
      </gnu:args>
    </gnu:configure>

    <gnu:check skip="yes"/>

  </gnu:package>

For those amongst you who know less about XML than I now unfortunately know,
the 'gnu:' bit is a namespace that identifies what schema these tags belong
to. The thing I like about this is that the particular schema for GNU
packages can be quite precise (though probably it should be slightly more
general than the example above). It's nice and clear what the above means (I
hope).

Thanks to the magic of XSLT, we can write a fairly simple transformation
program that converts the above into a standard Arusha config like the one
before.

But here's the cool bit: one of the problems with the Arusha config is that
it doesn't describe a GNU package, but a sidai-GNU package. You can see that
the config is "tainted" with a sidai'ism - the "PKGS_SRC" stuff. The
advantage of the specialised GNU spec is that it is untainted by this sort
of thing. So our XSLT transformer turns a "pure" GNU-style specification
into a sidai style one. With a different transformer we can turn it into
something completely different. So the sidai aspects can be abstracted out
into a single file that can be readily replaced with an alternative.

--- Stick with me here, because we're going places... ---

One of the goals of the original format - or if you prefer, one of the
axioms of Arusha - is that everything should be written down once, and only
once. Thus the Arusha schema is designed around a kind of document
inheritance model that encourages re-use. But if the actual specification is
machine generated, then there's no particular need for it to avoid
duplication. We could just as easily convert the GNU-style spec into a
complete list of everything that needs to be done to build and install a
GNU-style package at our site. The template instructions are only documented
once, in the XSLT transformer. In which case, we could throw out the
prototype model completely. I know prototypes are also used for
naming/grouping, but I'll come back to that.

So imagine that instead of Arusha's language being a super flexible
multiple-document model, it was instead a fairly simple list of instructions
along the lines of "execute the following command on the following machine".
The format would look sort of like a general sysadmin markup language.
Something along the lines of:

  <ark:script>

    ...

    <ark:step id="configure" state="keep">
      <ark:title>Configuring the package</ark:title>
      <ark:text>
        We configure Python to use the new memory allocater,
        the new cyclic garbage collector, and threads.
      </ark:text>
      <ark:shell onhosts="foo">
cd /item/ark-builds/python--2.2.1/sparc-solaris
./configure --with-pymalloc --with-cycle-gc --with-threads \
  --prefix=/._ark-deploy/python--2.2.1
      </ark:shell>
      <ark:shell onhosts="bar">
cd /item/ark-builds/python--2.2.1/rh-linux
./configure --with-pymalloc --with-cycle-gc --with-threads \
  --prefix=/._ark-deploy/python--2.2.1
      </ark:shell>
    </ark:step>

    ...

  </ark:script>

It should contain as few Arusha-isms as possible, and ideally no sidai-isms
at all. We could run this through a fairly simple transformer to generate a
web page explaining exactly how to install Python (or how Python *was*
installed).

Now we probably don't want to transform from a GNU-style package spec
straight to a complete list of instructions because that would involve
duplication. A vendor unpack-and-drop-into-place-style package would still
likely involve the same deployment and revelation mechanism that a GNU-style
package would at our site - we don't want to repeat ourselves in the XSLT
transformers. The trick is to use a hierarchy of schemas and transformers.

The GNU-style transformer would fill out the steps for building the package
and leave high-level instructions for deployment and revelation. Another
transformer then turns these into the actual deployment and revelation
steps. So perhaps the GNU-style spec would be transformed first into a
sidai-style package spec:

  <sidai:package name="python--2.2.1">

    <sidai:unpack>
      <sidai:source dir="OPENSRC">Python-2.2.1.tgz</sidai:source>
    </sidai:unpack>

    <sidai:link-farm once-per="build-type"/>

    <ark:step id="configure" state="keep">
      <ark:title>Configuring the package</ark:title>
      <ark:text>
        We configure Python to use the new memory allocater,
        the new cyclic garbage collector, and threads.
      </ark:text>
      <sidai:script once-per="build-host">
cd <sidai:package-build-dir/>
./configure --with-pymalloc --with-cycle-gc --with-threads \
  --prefix=<sidai:package-deploy-dir/>
      </sidai:script>
    </ark:step>

    <ark:step id="compile" state="keep">
      <ark:title>Compiling the package</ark:title>
      <ark:text>
        GNU packages can be built with a simple 'make all'.
      </ark:text>
      <sidai:script once-per="build-host">
cd <sidai:package-build-dir/>
make all
      </sidai:script>
    </ark:step>

    <sidai:deploy type="link"/>

    <sidai:reveal>
      <sidai:revelation-spec>
        ^lib/charset\.alias$           none
        ^share/locale/locale\.alias$   none
        .                              soft
      <sidai:revelation-spec>
    </sidai:reveal>

  </sidai:package>

We make use of namespaces to freely mix together elements of the underlying
'ark' language and the specialised 'sidai' language. Note that the scripts
themselves are "parameterised" on a few sidai concepts like the build and
deploy directories. All of the 'sidai:' tags will be re-written in the next
step, which is to run this spec through a 'sidai->ark' XSLT transformer.

Now you might be thinking, "I don't particularly want to learn a lot of XSLT
in order to change a simple configuration at my site - I've seen how
monstrous these transformers can get!". But fear not. Conveniently, XSLT
programs can open other documents than the one they are transforming and
retrieve bits from them. So the sidai maintainers would write the sidai
transformer to look up site-specific data from a "sidai config file" - I
guess something like the 'team.xml' file from before, but more focused since
bits of configuration actually end up spread across the 'team.xml' and
various 'ALL.xml' files.

-- The end is nigh! ---

So, to recap: we have a bunch of specialised schemas for particular kinds of
package (like GNU, vendor-splat, perl-module); we transform these into a
single site policy schema (for instance sidai); we then transform the site
schema into the actual Arusha sysadmin scripting language - this language
can either be executed by the Arusha engine, or turned into documentation.

The high-level specifications can now be freely used across sites that can
have wildly different underlying mechanisms for carrying them out - as long
as we all agree on what a GNU-style package is. If you're happy with the
sidai policy then you can just keep the 'GNU->sidai' transformer. If you
want to change how deployment and revelation are done then you edit the
'sidai->ark' transformer. But another site might have 'GNU->cfengine' and
'cfengine->ark' transformers.

Concerns:

 1) Is XSLT expressive enough? - Other than being a functional language,
    most XSLT engines conveniently allow you to register your own
    external function libraries and call into them from a transformer.
    So you could still put all the hard stuff in Python and then just
    make a 'py:sidaiDeploy' call.

 2) It can't be that simple - It probably isn't. I'm sure I'm missing
    lots here, but that's where you guys come in!

 3) What about the grouping side of prototypes? - I'm glad you asked that
    question...

--- Perhaps all things are not created equal after all... ---

I never really found a whole heap of use for grouping of packages. I've
rarely decided to reveal all GNU packages together. Generally I'm running
everything to bring up a new box, or I'm working on one package.

In terms of hosts, there's much more reason to have grouping. But here's
where I recant from an earlier position: my new position is that the host
specs are *very different* to package specs. They would just contain
configuration information used by the transformers and the Arusha engine. So
they would have a completely different schema to the other spec files.
Exactly what that is I haven't thought about yet. I have vague ideas running
around my head about special-purpose schemas for particular kinds of host
(like server and workstation schemas) and transformers that turn these into
Arusha host specs, but I'm pretty woolly on all that still.

Interestingly, if we think about the team repositories as now being a kind
of XML database, then we could think of the 'package' directory as being a
"collection". In which case one can use the XPath language to formulate
queries into this collection in standard XML:DB style. For instance a query
like:

    /gnu:package

would refer to all of the GNU packages. So if I really did want to reveal
all of the GNU packages together, perhaps I could use an XPath query. XPath
provides lots of other cool possibilities like:

    /gnu:package/gnu:check[@skip="yes"]/..

would refer to all of the GNU packages which skip the check method. Though I
can see lots more uses for XPath for getting information out of host specs,
for example:

    /host[normalize-space(type) = "solaris"]/@name

might pull out the names of all of the solaris boxen. This is the sort of
thing I would expect to see happening in the XSLT transformers that build
Arusha scripts.

So having argued in the past for rationalising and extending the Arusha
object model, I'm now arguing for throwing it away entirely ;-)

I'd be very interested to hear any and all comments on these ideas.

*phew*

Jonathan

-- 
jonathan hogg, one good idea ltd, 131 queen margaret dr., glasgow g20 8pd
http://www.onegoodidea.com/ tel:+44-(0)7976-614338 fax:+44-(0)7970-537451



-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31