Re: circumflex/caret --> scoped crm114 script variables

On Wed, Feb 25, 2009 at 3:54 PM, Bill Yerazunis <[email protected]> wrote:
>   WARNING: the latest crm114-GerH supports scoped script variables, so a
>   particular (ugly) variable instantiation use in scripts won't work any
>   longer: see
>
>   http://hebbut.net/Public.Offerings/crm114/crm114-BlameBarack-Branch-preliminary.html#scoped_variables
>
>
> Whoa!!!!  Interesting!
>
> I tried something like that but got tied up in knots.  For example:
>
>  isolate (:a:) /Foo!/        # a is of global scope
>  call /:capture_func:/
>  output /:a: is :*:a:\n/
>
>  :capture_func:
>  isolate (:z:) /abcdefghijklmnop/           # make :z: be local
>  match [:z:] /...(...).../  ( :: :a: )      # capture :a: into :z:
>  return
>
> Now, :a: is attached to the 4th through 6th characters of :z: - but
> :z: is now _going away_.  So, what does :a: have?

What you'd expect: 'def'.
Hm, nice extra test case, BTW.


> Second question: are these scoped variables "dynamic" or "static" -
> that is, if I call a function with a locally scoped variable a second
> time, what is the value of a locally scoped variable?  Is it retained
> between calls (i.e. "static", as in C), or is the locally scoped
> variable reinitialized from scratch (so DEFAULT values values will
> trigger in!)

See that URL up there ;-)

The current release has them as 'static' but that is considered a bug,
NOT a feature.


Note that I come from the other direction at this: I got absolutely
fed up with having to keep track which bit of code 'isolate'd or
otherwise instanciated variables, which would then magically appear
everywhere. This was quite nauseating while I worked on
mailfilter/mailreaver/mailtrainer/maillib, as those have nontrivial
LOC counts, so the lack of _scope_ in this script language started to
seriously hurt.

When you've had a look at the latest few GerH releases, you'll find
that mail*.crm has undergone some changes, one part feature
unification, one part extra checking and Windows portability (path
conversion and such stuff) and one last part: kicking out the need for
syscall. It's got two drawbacks:
- it's slow (see the timing tests in GerH tests/ )
- it's very system dependent (there's no such thing as a bang line in
Windows and you can forget about running /usr/local/bin/crm114 as well
;-) )

And syscall is not needed after all: mailreaver can load maillib and
so can it load mailtrainer and call it's relevant code as a
subroutine....... if only we had scoped variables, for otherwise the
'magick' regarding sudden changes to existing variables becomes
unbearable.

>From a language design point of view, the scoping rules should come as
close to other, more well known languages, as possible.
So all this static, dynamic, etc. is all very nice, but let's be real:
serious software engineers know globals (and statics are just another
form of global) are Bad Form. The coding standards abound which forbid
this type of variable declaration entirely (and I'm not _that_ anal
(close, baby, _close_, though) and using reduced scope (local scope)
for your variables has been advocated worldwide for, what, 30? 40?
years?
Local scoped variables allow one to use Sound Coding Practices(tm) --
something crm114 can use in several areas ;-)

Anyway, the code tonight just passed several tests. Screw 'static'
scope, not gonna happen. Not on my watch, anyway.

So we have local scope (what you call 'dynamic'), the old fash 'global
scope' and a particular called 'outer scope'.

Let me explain: when variables are instantiated (by isolate or other
operations), by default, these variables are instantiated in 'local
scope'.
'Scope' is defined as the call depth, i.e. you enter a new (sub)scope
when you execute a 'call' operation and you 'pop' to the next 'outer
scope' when you execute a 'return' operation.

Notes here: 'trap' and the lot do NOT change scope. NOR do curly
braces (too bad for the C lovers; crm114 script isn't exactly 'C', so
I felt free to disregard that language for this.)


Then how do you access outer scope/global scope variables?
Where 'outer scope' is, for the reference point of the currently
active call depth (= scope) is just like 'global scope'.
Simple: when you execute commands which assume variables exist, such
variables are sought in local scope, and when not found there, the
outer scope levels are ascended until we either find the variable name
or run out of scope levels.
That means that your :a: up there is found, and it sits in the first
outer scope level, from the point of view of :capture_func:

Any edits to any variables are performed (and thus visible) at the
scope where they were instantiated. (So the edit made by the 'match'
to that ':a:' up there, will still be visible when :capture_func:
'return's.

Variable instantiation: by default 'isolate' instantiates a new
variable instance at local scope (if one doesn't exist already). Note
the 'by default' because there are a few exceptions to this rule:

isolate <default> (:var:) /.../

essentially is just another operation which *references* a variable --
due to the <default> -- , just like 'match', 'eval', etc.etc., so the
general rule applies there: the variable is looked up in local scope,
when not there, outer scopes are ascended until either found or we run
out of scopes.
When 'isolate <default> ...' then decides to instantiate the variable,
as it doesn't exist in any accessible scope, the 'instantiation' rule
applies: variables are instantiated BY DEFAULT in local scope.

So.... you'd love to instantiate a global variable, eh? Easy. Add
<keep> as attribute to the operation and you're switching to the
'augmented rule' for variable instantiation:
with <keep>, the variable is instantiated (WHEN it has to be
instantiated!) at GLOBAL scope (that's the top-most 'outer scope')

The following commands support <keep> from about a hour ago:

isolate
call   (the return variable!)
match  (the subexpr variables!)
union
intersect
eval   (the result var)
input

Note that eval, match, etc. often do not instantiate the variables
they fill with results, but they CAN, also in vanilla. Here, we just
add the possibility to have them instantiate the variable in 'topmost
outer scope' a.k.a. 'global scope'.

By sticking to either 'local scope' and 'outer scope' rules like that
the amount of detail you need to remember/learn when programming in
this language is kept minimal. And I'm a big fan when it comes to
smooching, especially when it comes to language design.

so

isolate <keep> (:a:)

will create a 'global scope' instance. Note the rules as described
above: IFF the variable is not already visible in the current scope
(by existing in the current=local scope or any of the visible outer
scopes.

This design allows to write structured code (which you should do). Use
'isolate' as the crm114 script equivalent of local scope variable
declarations as you know them from other languages. More precisely,
you should use this construct for that:

isolate (:var:) /.../

and don't 'forget' the /.../ -- like Bill did in several spots in his
scripts -- because in vanilla, GerH-old style and GerH-as-of-today all
treat
  isolate (:var:)
ever so slightly different from
  isolate (:var:) /.../
and
  isolate <default> (:var:) /.../



> Third question: if it's really scoped, then there should be a way
> to create that local variable as an overlay, so it _keeps_ it's
> value on return, like this
>
>   isolate (:a:) /foo/
>   call /:capture_func:
>   output /:*:a:/         # should be back to /foo/
>
>   :capture_func:
>   isolate (:a:) /bar/
>   output /:a:/
>   return

See above: <keep>'s your answer:

>   :capture_func:
>   isolate <keep> (:a:) /bar/
>   output /:a:/
>   return

>

> This drove me to consider that perhaps we needed four different waus
> for a variable to be created:
>
>   isolate   - global scope
>   dynamic   - local scope, state lost on RETURN
>   static    - local scope, retained state
>   match     - global by default, but will flow with dynamic or static.

Bill, please, I like complexity, but NOT in language design. Treat all
as equal: match, eval, insolate, anybody who (optionally) instantiates
variables does so at a given scope, which should be local scope as any
developer coming from other languages expects /that/; not some
fancy-oh-my-need-to-remember-this-exception-regarding-match-here. Yes,
that gave me hackles.


And yes, it breaks some crm114 scripts, but then my thoughts on those
are that they aren't exactly the pinnacle of coding excellence either
and shouldn't serve as examples how to write crm114 script software
for anyone.

Most notably, the load_cfg code in maillib croaks. A simple <keep> for
that isolate in there fixes things up nicely. Which is a hack.
The alternative is instantiating the variables up front in outer-most
scope, where they are expected to exist after all by the remainder of
the mail*.crm code. See the current GerH for that change. Though,
granted, plonking <keep> in there is less lines. So I'm thinking of
going with that for the next release, just to prove a point. Maybe.

Oh, and have a gander at the latest release anyway: mailfilter.cf also
supports [] next to // as value delimiters, of course. It's not just
isolate :-)






> In which case, doing a DYNAMIC or STATIC on a MATCHed variable
> does... what?  Do they create local overlays, in which case the old
> (MATCHed) value is restored on RETURN, or does it change the type of
> variable so they're erased (for dynamic) or hidden (for static) on
> return?

See above.


> And if it's a MATCH captured variable, are changes made while in the
> subroutine kept in the outer variable, or rolled back on RETURN?

Changes are kept... up to a point. Since you are suggesting something
here which simply is not a good coding practice anyway (but very nice
to have around for when we organize the first international OCCC
(Obfuscated CRM114 Coding Contest)) the details are a little hairy
there:

upon return, such variables point to data which is still valid -- in a
way. The fact is that the current implementation 'flags' variables
which go out of scope as 'out of scope'.
When you enter that scope level again (or a deeper scope depth level)
and attempt to instantiate a NEW instance with that same variable
name, the out-of-scope variable gets re-used (without passing on any
old, now definitely out-of-scope, content).
So, in short: yes, the variable will be valid. As a rule of thumb, you
can assume it becomes 'disputable' / 'invalid' when another 'call'
operation enters that scope depth level again.


> These kinds of questions are why there's not been a mainline release
> with that functionality.  The semantics are not trivial.

Twas simpler than you might think ;-)

And to answer Paolo at the same time: no extra memory cost whatsoever.
(Unless you count a few integers per variable as major storage cost --
which it isn't: the increase is maybe, hmmm.... 16 bytes per variable
instance total. And out-of-scope variable storage is re-used (when you
use the same name in an active scope again), so performance is quite
comparable to the global-only vanilla or previous GerH releases. :-)


> But your notions make me think more clearly on this again..  (as well
> as rethinking multiple-values-return capability... which is yet
> another matter, but simpler than this one at least)

Which is about dialing up the hardcoded limits on the delims. Instead
of s1start/s1end, etc., go s_start[0]/s_end[0], etc.: array notation
in C and you either statically or dynamically allocate such arrays
when parsing the script. But that's for another day.

I would get really worried when folks start reporting crossing the
100KLOC boundary in crm114 script. Because then, it's become another
JavaScript gone bursar while getting nada for the dried frogs pills
shipment delivery date.


-- 
Met vriendelijke groeten / Best regards,

Ger Hobbelt


(mail*.crm surpassed the 5 KLOC barrier already a while ago... quite
an achievement for a language which was, IMHO, a nice 'geek exercise'.
[I recall a similar thing I did way back when - the idea that you can
write an entire interpreter in yacc, no ASTs, no P-code, nothing. Just
a few lexer hacks and the yacc/bison state machine. And it worked. It
worked great, even. But in some ways it was _very_ much like the
crm114 language, emotion- and common engineering sense-wise.])



--------------------------------------------------
web:    http://www.hobbelt.com/
        http://www.hebbut.net/
mail:   [email protected]
mobile: +31-6-11 120 978
--------------------------------------------------

------------------------------------------------------------------------------
Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, CA
-OSBC tackles the biggest issue in open source: Open Sourcing the Enterprise
-Strategies to boost innovation and cut costs with open source participation
-Receive a $600 discount off the registration fee with the source code: SFAD
http://p.sf.net/sfu/XcvMzF8H