More on meta-ness (long)

Hi again,

In my Copious Free Time, I've been continuing to play with my demo
Arusha-TNG code. I've been working on making things more orthogonal. In
particular, the code execution mechanism and the current specialness of host
things.

My starting point was the 'once-per' magic. If one can say that some code
should be executed for, or by, every host, shouldn't one be able to say that
some code should be executed for, or by, every member of any particular
thing type?

So I've got a new syntax for 'once-per':

    <foo>
        <code once-per="host:ALL"> ... </code>
    </foo>

means this code should be executed on every host, i.e., hosts that inherit
from 'ALL'. The way this works is that when I ask for the 'foo' field of the
thing (whatever it is) I get back a sort of bound-method object. When this
object is called, it gets a list of all of the host things and then calls
the 'execute_field' method on each in turn.

Since 'execute_field' is a ArkHost class method, it can be supplied from the
host meta-thing. Here's a short example of the method I'm using in the host
meta-thing:

    <execute-field>
        <!-- Inherits parameters: name, forthing, lang, code, params -->
        <code><![CDATA[
lang = lang and lang.lower() or 'sh'
getattr(self, 'execute_' + lang)( name=name, forthing=forthing, code=code,
                                  params=params )
        ]]></code>
    </execute-field>

This method is passed some parameters from the field. It gets the name of
the field, the thing the field came from, the language the code is in, the
code string itself, and the built-up parameter dictionary. My little code
snippet above defaults the language to 'sh' (note that the default language
is decided by the thing that has been asked to execute the field), and then
calls itself again on an appropriate method to execute the language
specified.

So here's an example for a particular language:

    <execute-sh>
        <param name="name"/>
        <param name="forthing"/>
        <param name="code"/>
        <param name="params"/>
        <code><![CDATA[
script = ''
for key, value in params.items():
    script += '%s=%s\n' % (key, `value`)
script += code.strip() + '\n'
self.run_command( command=self.SH(), input=script )
        ]]></code>
    </execute-sh>

This simply builds a shell script and then calls itself again on the
'run_command' method, passing in the location of SH on this host and the
script as input. The 'run_command' method looks like:

    <run-command>
        <param name="command"/>
        <param name="input"/>
        <code><![CDATA[
run_host = self.run_host()
if self is not run_host:
    command = '%s %s %s' % ( run_host.SSH(), self.hostname(), command )
print '--- Executing %s ---' % command
import os
pipe = os.popen( command, 'w' )
pipe.write( input )
pipe.close()
print '------'
        ]]></code>
    </run-command>

The first thing this method does is try to work out if we are on the host we
want to execute the code on. The 'run_host' method, which I'll omit, simply
calls gethostname and then tries to find a host thing to match this. If
we're on the wrong host, then it finds the location of SSH on the run-host
and amends the command to indirect via this. Then it does the obvious popen
stuff to execute the command and pipe in the input.

An interesting feature of all of this is that language support is a feature
of the thing asked to execute the code. So hosts support a certain set of
languages, but other things might support different ones. One of my goals
with this is to be able to add new types of thing, such as a router where
the meta-thing execute_field would telnet to the box, go through a login,
and then just send it the code supplied, e.g.:

    <router name="cisco">
        <telnet-username> ... </telnet-username>
        <telnet-password> ... </telnet-password>
        <enable-password> ... </enable-password>
        <reset><code privileges="enabled">
reload
        </code></reset>
    </router>

Note that if you don't specify a once-per then the field calls execute_field
on the thing itself. My plan is also that privileges information is passed
by the field to the thing that is executing it. The target thing can then
decide how to interpret the privileges and what it needs to do to gain them.
But with the above, one could then issue commands like:

    % ark router reset gateway1

So what else needs to change? Well the proxy mechanism and the host
constraints stuff. The proxy mechanism seems pretty easy. My thought was to
have something along the lines of:

    <code once-per="host:ALL" proxy="host:slimy"> ... </code>

to specify a single proxy for all calls. This would then call the
execute_field method on the slimy host thing once with a list of everything
it's proxying for. Also:

    <code once-per="host:ALL" proxy-table="build-hosts"> ... </code>

would look for a field with a table value that maps from host prototypes to
the actual proxies. Something like:

    <build-hosts><table>
        <entry name="solaris"> host:slimy </entry>
        <entry name="linux"> host:slither </entry>
    </table></build-hosts>

An important thing to note here is that the proxies specify the thing type
as well as name. This is because I want to be able to have things like a
package for router configs that has a field that should be executed once-per
router, but which has the actual code executed by host proxies, e.g.:

    <package name="router-config">
        <trusted-hosts><table>
            <entry name="external"> host:trusted1 </entry>
            <entry name="internal"> host:trusted2 </entry>
        </table></trusted-hosts>
        <install><code once-per="router:ALL"
                       proxy_table="trusted-hosts" lang="sh">
for router in $proxy_for
do
    # Upload generated router configuration...
done
        </code></install>
    </package>

Which just leaves host constraints. Since code is not necessarily being
executed by a host-thing, these need to be made more general. There are up
to three different things in play: the original thing (e.g., a package), the
once-per things (routers), and the proxy (host). I would guess that it would
only be useful to constrain on the latter two. I'm not sure exactly how to
specify these, but perhaps:

    <constraints>
        <router spec="external">
        <proxy spec="trusted2">
    </constraints>

The first constraint is named after the type of the once-per things. The
second constrains the field to only apply on a particular proxy (or range
with a prototype spec). Such constraints are only really useful on
overriding fields.

So far I've got the 'once-per' stuff working in my demo code and have
working examples of hosts doing local and remote execution of sh and perl.
The proxy stuff is my next target. I don't have any kind of contraints
framework, so I'll leave that for a bit.

Comments on all of the above welcome. Am I going in the right direction?
Does anyone really care?

Cheers,

Jonathan

-- 
jonathan hogg, one good idea ltd, 131 queen margaret dr., glasgow g20 8pd
http://www.onegoodidea.com/ tel:+44-(0)7976-614338 fax:+44-(0)7970-537451