Re: [Infrastructures] late thoughts on turing paper - when not to have perfect replay?

On Sun, 13 Jul 2003 [email protected] wrote:

> The "work lists" you're describing above are the way I've always thought
> of this, without using those words.  I guess the main difference between
> host classes and work lists is that host classes are usually implemented
> as some abstract, maybe even inheritable thing with attributes that can
> be twiddled as if the class were a black box, while work lists are just
> dumb ordered lists of commands to be run, usually without much in the
> way of conditionals, where the sysadmin has to understand the line-item
> detail in order to make much use of them at all.

I think there's still confusion here.  Let's see if I can draw a map:

Isconf 2:  All lists of work to do are attached to actual host instances.
Instances can derive from other instances, but the old instance can no
longer be modified in any way.  For instance:

host1:
   stuff1
   stuff2
   stuff3

host2:
   host1
   stuff4
   stuff5

If you go back later and modify 'host1', then you are breaking the rules
of the system (although the tool in no way attempted to stop you from
doing this).  Note also (and this is what I initially disliked) that there
is no way other than naming to differentiate a host from a target in a
host's work list:  host1 and stuff4 are treated equivalantly, even though
they are functionally quite different.

Isconf 3:  All host types are abstract and can be subclassed, and all
hosts can be as many host classes as you want:

# types
type1:
   stuff1
   stuff2
   stuff3

type2: type1
   stuff4
   stuff5

type3: type1
   stuff6
   stuff7

-------
# hosts
host1: type2
host2: type2 type3

Notice that it is immediately obvious whether you are operating on a host,
a type, or a make target, because they are all treated very differently;
this was my main goal with this typing.  I also wanted the ability to
modify base types while retaining ordering, which is why I added
timestamps.  In Isconf2, all of its work was done before any subclasses,
whereas in Isconf3 work is done only according to its timestamps.  This
allows me to make the following type of host structure:

osbase:
   initial_fileset

sunos_base: osbase
   sun_specific_stuff

hpux_base: osbase
   hpux_specific_stuff

Now if there's something that I want to do to every system on my network,
I add it to 'osbase'; if I only want to do it to Sun machines, I add it to
'sunos_base'.  In other words, I can take my heirarchy and pick the point
of specificity.

Now, moving on to what I'm talking about WRT just having ordered work
lists:

osbase:
   dostuff 01
   install_cfengine 13

type1: osbase
   morestuff 02
   cluster_setup 14

type2: osbase
   yaystuff 03

install_cfengine:
   install_openssl 03
   install_bdb 04
   install_cfengine 04
   cfkey 05
   cfupdate 06
   cfinit 07
   cfstart 08

cluster_setup:
   install_vcs 09
   install_cluster_fileset 10
   setup_cluster_devices 11
   start_cluster 12

This is obviously an unrealistically simple heirarchy, but you get the
point.  I am not going to create any "install_cfengine" types, nor any
"cluster_setup" types, yet it's worth creating them as completely separate
work lists because they are related.  If I just included them in the
appropriate host types, then it wouldn't necessarily be obvious that the
steps are related in some way; this becomes especially true as you add to
the work list, because related items can be quite far apart in the work
stack.  Here's a list of items for just one of my node host types (meaning
it is not subclassed in any way, so it is quite specific):

rsync_server: sun_server
    rsync/fileset__code__isconf--code
    rsync/fileset__filesets__isconf--filesets
    rsync/add__code
    rsync/add__filesets
    filesets-coderoot
    filesets-filesetroot
    known_hosts_cron
    rsync/startup
    ln/..--init.d--rsyncd__etc--rc2.d
    /etc/rsyncd.conf
    rsync/fileset__isconf__isconf--code--isconf
    rsync/add__isconf
    filesets-isconfperl
    filesets-perlparser
    hourly_cron
    rm/norecurse--etc__rc2.d__rsyncd
    ln/..--init.d--rsyncd__etc--rc2.d--S80rsyncd
    rm-badlinks
    ln/etc--init.d--rsyncd__etc--rc2.d--S80rsyncd
    installcopy/copy--openssl--0.9.6g
    installcopy/copy--db--3.3.11
    installcopy/copy--cfengine--2.0.4
    installcopy/cfenginelinks

Can you pick out which items above are related?  Every item in that list
is dedicated entirely to building an rsync server, except the last four
items.  Those four items are some of the first steps necessary to install
cfengine.  The leap I'm talking about would take those four items and move
them into their own list, and then let me say:

rsync_server: sun_server cfengine_setup

This is a bit confusing, because it would look like the cfengine_setup
work would run before the rsync_server work, but the timestamps will make
sure it doesn't.

In other words, Steve, I haven't gone back to the more simplistic approach
of Isconf2, I've moved even further away from it.  And with my integration
with cfengine, I'm going to begin attaching work lists to cfengine classes
that don't exist at all as ISconf types.  I would no longer need to list
hosts as either "sunos" or "hpux" in ISconf; instead, cfengine would
auto-discover that and ISconf would just do the right thing.

This could save quite a bit of trouble, because it would become very
difficult to accidentally enable the wrong platform on a host (which I've
had coworkers do before).

> Every time I've tried to make more complicated tools I've wound up right
> back with this same dirt-simple model -- work lists.  The turing paper
> claims that these work lists have to be fully ordered in order to be
> 100% reliable.  In practice, we all know that the best they'll ever be
> is partially ordered, overlapping, and so on.

Heh, every time I try to create a simple tool, I end up creating a far
more complicated one. :)

> You said that work lists shouldn't overlap -- I betcha they always will.
> I think what you did with the timestamps in v3 was a stroke of
> appropriate genius in solving that problem, and something like it is
> probably always going to be a feature of any tool that attempts to make
> reproducible changes.

If I continue to use any semblance of ISconf in my future work, it will
probably morph entirely into either a cfengine or psgconf (or both)
module, and I'll just use a flat file as my stamps database.  If cfengine
actually develops a real module interface, one which allows modules to see
cfengine variables, then I could easily get rid of the hosts file, and
possibly even the types file (although that might be more effort than it's
worth).

Luke

-- 
Of the thirty-six ways of avoiding disaster, running away is best.
                                             -- Chinese Proverb


-------------------------------------------------------
This SF.Net email sponsored by: Parasoft
Error proof Web apps, automate testing & more.
Download & eval WebKing and get a free book.
www.parasoft.com/bulletproofapps1