disruptive change re "proxy hosts"

Folks, a little chat has taken place (regrettably off-list
:-() about some inadequacy in the ARK "proxy host"
mechanism.  This message is to put some of the discussion on
the record.  Recall that when you say

   ark package compile ALL

you mean, "For every package derived from the 'ALL'
proto-package, run its 'compile' method *on every host*".
The same would be true even if the 'thing' were a vendor,
user, support call, or whatever.  (In other words, by
default, every action is a "site-at-a-time" action.)

That may be the meaning, but it's a silly thing to do.  If
you have a 100 Suns, and you're compiling apache--1.3.20,
you wouldn't want do it 100 times.

Essentially, you want *one*  machine to do the compile,
acting as a *proxy* for all of the others.  Similarly when
you unpack a tarball, patch some source, and so on...

But trouble was lurking in the darker recesses of this
feature... Bits of the "conversation" are below, and we seem
mostly in agreement.  *Yell* if you have any problem with
it, or spot some even-darker-recess we've missed.

This change is disruptive, in that .xml files need lots of
changes; however, it seems to be just three M-x
tags-query-replace's in Emacs, so my current plan is *not*
to bump the xml-version numbers.

(Expect code later today.)

Will

=== [most of a message (by Matt) that explains things]

At the moment, the proxyable thing not *only* determines
which host the method is run on, but also the granularity by
which it has to be run (ie.  once per host, once per
platform, once per site...). I think these two factors need
to be separated. In addition, the proxyable attribute
affects the *context* in which the method is evaluated, for
example when processing @host:FOO@ style macros.

In my case, what I wanted to do was to generate config files
for a new host of a new platform type, which I could then
manually copy to the new host to bootstrap the
ark-ification. However, this turned out impossible, because
the compile methods were running once per platform (with a
call to sidaiTemplatePerHost to generate a file for each
host), and I didn't have an existing instance of that
platform to proxy the build onto.

One way around this would be to make hosts-supported 'ALL'
only, or proxyable="." for the method, which would make the
method run once per site. The method itself would then
generate files for the new host when it iterated through all
the hosts in sidaiTemplatePerHost.

However:

1) This is bad, because we have two mechanisms for
specifying the granularity of the method; one by using
"proxyable" and "hosts-supported", and the other by manually
iterating over a list of hosts *within* the method
(eg. sidaiTemplatePerHost). The second mechanisms of course
does not make any record of this fact in the state.

2) The context in which the method is evaluated is that of
the run host, rather than the destination host. This means
that any parameters that are used to generate the config
file and make use of @host:FOO@ resources will be wrong. Of
course some parameters will be correct, for instance to find
out the location of a particular command on the run-host....

3) Another problem, that Will mentioned to me with doing the
'hosts-supported=ALL' thing, but I can't remember ;)

What I'm proposing is this (and it's quite complex, sorry):

1) Separate out granularity of execution and
proxyability. Then we can ditch "sidaiTemplatePerHost" type
things and do configuration file generation "once per host"
with the engine. This actually simplifies things and gives
us one mechanism. We can actually run the method on a single
proxy host for entire site, but have it executed on *behalf*
of every host.

2) State entries should refer to the host we are running the
method on behalf of, not the proxy.

3) Make a distinction in parameterisation and host-specs,
between the host the method is being run for, and the proxy
on which it is actually being run.

4) (as an aside). Make proxyable and granularity elements so
they can be overridden at a higher level of the prototype
tree.

To make this clearer, here is an simple example, for
generating /etc/resolv.conf:

<compile>

<!-- proxy host parameters -->
<param name="CAT"      value="@host:proxy:CAT@"/>
<param name="MKDIR_P"  value="@host:proxy:MKDIR_P@"/>

<!-- actual host parameters -->
<param name="DOMAIN"   value="@host:actual:domain@" />
<param name="NS_PRI"   value="@host:actual:primary-name-server@" />
<param name="NS_SEC"   value="@host:actual:secondary-name-server@" />
<param name="hostname" value="!actual_host.idString" />

<granularity>once-per-host</granularity>
<proxyable>slimy</proxyable>

<code lang="sh"><![CDATA[
cd $BUILD_DIR
$MKDIR_P ./$hostname
cd $hostname

$CAT >resolv.conf <<EOF
domain $DOMAIN
nameserver $NS_PRI
nameserver $NS_SEC
EOF

]]></code>
</compile>

=== [Will's first shot at a reply]

So, let me think, what's my (slight) counter-proposal:

* code attribute 'granularity' can have values:

  'once-per-host' [DEFAULT]

  'once-per-site'

  'hosts-supported' : once for each in a list of
    (proto-)hosts in list
    <hosts-supported> minus those in <hosts-not-supported>

  'something-else' : once for each in a list of
    (proto-)hosts in list <something-else>

* code attribute 'proxy-hosts' can have values:

  'no' : can't use a proxy;
	the default if granularity="once-per-host"

  '.' : any old host (will choose current run host);
	the default if granularity="once-per-site"

  'some-table' : use <some-table> of (proto-host, real-host)
     pairs to find the real host;
	if granularity="hosts-supported", then the default
	is proxy-hosts="proxy-hosts"

  If granularity="something-else", there is *no* default for
  proxy-hosts.

=== [Will, getting down to specific sytax]

Concrete syntax things... I'm currently going for

<code once-per="(host|site|hosts-supported|other)"
      proxy-hosts="(none|.|some-table)" />

In the macro language

     @host:FOO@ and @proxy-host:FOO@

And in the constraints language

     <host-spec> ... </host-spec>
     <proxy-host-spec> ... </proxy-host-spec>

[Matt subsequently showed that <proxy-host-spec> is
absolute utter piffle.]

Do you think we're ever going to have comparable mechanism
for proxy-users, proxy-vendors, proxy-web-sites (i.e. other
things)?

=== the end