Defining variables in the caller's environment (was: ERR_RETURN ignored in 'if' in a source'd file [FIXED])

This is a reply to James' message. I changed the subject because the
content could be useful to a wider audience.

3 points:
- Thanks
- Techniques to define variables in the caller's environment
- Zsh feature requests (Bart, wdyt?)

*Thanks*

Many thanks to Philippe Altherr for all the ERR_RETURN/ERR_EXIT fixes, and
> also, thanks to all maintainers!


Thanks for your thanks 😊 I remember that the whole thing turned out to be
much more involved than initially thought. For sure others contributed too,
in particular Bart!

Iirc, the fix for "eval" and "source" was just a side outcome of the bigger
change. Great to see that it actually helped someone. Also thanks for
sharing your use case. It made me realise that I could improve one of my
techniques to define variables in the caller's environment, which brings us
to my next point.

*Techniques to define variables in the caller's environment*

Defining variables in the caller's environment is a recurring issue that I
face in Zsh. It's not something that is explicitly supported but there are
techniques to achieve it. I mainly rely on two.

My first technique makes use of the EXIT trap. When an exit TRAP is
configured within a function it triggers when the function exits and runs
the trap code in the *function's caller* environment. The following proof
of concept implements the equivalent of local name=value.

function define-var() {
  trap "local $1=${(q)2}" EXIT
}
function main () {
  define-var var "Hello world!"
  echo "var=$var"
}
% main
var=Hello world!
% echo "var=${var:-<no such global variable>}"
var=<no such global variable>

The technique has the advantage that the helper function can define its own
local variables, for example to compute the initialization value of the new
variable, without any risk to corrupt the caller's environment; only the
variables defined in the trap code will be added to the caller's
environment. The main disadvantage is that generating the trap code can
quickly become much more complicated if more than one variable need to be
defined and/or their initialization values aren't simply strings but
arrays. Things are even worse if the set of variables to define depends on
parameters of the helper function. The use of the EXIT trap also looks, and
arguably is, very hackish.

My second technique makes use of the Zsh associative variable functions
that gives access to the code of all defined functions.

function define-vars() {
  local var1=Hello
  local var2=Bye
}
function main () {
  eval $functions[define-vars]
  echo "var1=$var1 var2=$var2"
}
% main
var1=Hello var2=Bye
% echo "var1=${var1:-<no such global variable>} var2=${var2:-<no such
global variable>}"
var1=<no such global variable> var2=<no such global variable>

The technique is particularly useful when many variables need to be
defined. In this case all locally defined variables end up in the caller's
environment. Therefore the technique isn't ideal if local variables are
needed for local computations as they may accidentally corrupt the caller's
environment.

If arguments need to be passed to the helper function to compute the names
and/or the values of the variables, it's possible to do so but it's a
little cumbersome.

function define-vars() {
  local var1=$1
  local var2=$2
}
function main () {
  local args=("$@")
  set - Hello Bye; eval $functions[define-vars]
  set - "${args[@]}"
  echo "var1=$var1 var2=$var2"
}
% main
var1=Hello var2=Bye

If there is no need to save and restore the positional parameters, it's not
too bad but the "call" to the helper function looks weird.

The first thing I realized with James' message is that I can achieve the
same by relying on a helper script instead of a helper function. Indeed,
scripts called with the . and the source builtins run in the caller's
environment.

{ echo 'local var1=$1'; echo 'local var2=$2' } > define-vars.zsh
function main () {
  . ./define-vars.zsh Hello Bye
  echo "var1=$var1 var2=$var2"
}
% main
var1=Hello var2=Bye
% echo "var1=${var1:-<no such global variable>} var2=${var2:-<no such
global variable>}"
var1=<no such global variable> var2=<no such global variable>

This has the added advantage that passing arguments is much more natural.
However, it has the disadvange that part of the code lives in a separate
file.

The second thing I relaize with James' message is that I could combine his
and my technique to benefit from the best of both worlds.

function define-vars() {
  local var1=$1
  local var2=$2
}
alias define-vars='. =(<<<$functions[define-vars])'
function main () {
  define-vars Hello Bye
  echo "var1=$var1 var2=$var2"
}
% main
var1=Hello var2=Bye

Thanks to the alias, it's almost perfect. I say almost because behind the
scene, each call to define-vars triggers the printing of its code and its
parsing by the . builtin.

What if instead the helper function could be called as follows?

. define-vars Hello Bye

This is actually possible in ksh and it runs the called function in the
caller's environment! Which brings us to my last point.

*Zsh feature requests*

3 feature requests:
- Dot notation à la ksh to call functions
- Named references that set variables in upper scopes
- Typeset option to set variables in upper scopes

*Dot notation à la ksh to call functions*

In ksh, the dot notation allows to call functions such that they run in the
caller's environment. As demonstrated above, this could also be useful in
Zsh. Any reason why this could not be done?

If the invoked entity exists both as a function and as a script in the
PATH, then ksh prefers the function. For backward compatibitly Zsh may have
to do the opposite.

*Named references that set variables in upper scopes*

In my first technique, I rely on the EXIT trap to define the variable. With
the upcoming next release of Zsh, I could instead almost rely on named
references.

function define-var() {
  typeset -n -u ref=$1
  ref=$2
}
function main () {
  define-var var "Hello world!"
  echo "var=$var"
}
% main
var=Hello world!
% echo "var=${var:-<no such global variable>}"
var=Hello world!

I say almost because unfortunately assignments to named references to not
yet defined variables always define the variable in the global scope.
Adding an option to specify that the variable should instead be defined in
the enclosing scope (for -u named references) shouldn't be technically
difficult. The feature would however break a long standing invariant,
namely that no variables can be defined in enclosing scopes. For that
reason, it's not something that I would consider adding in the upcoming
release but I think it would be worth considering it for the release after
it.

*Typeset option to set variables in upper scopes*

If in a future release named references are enhanced to be able to define
variables in upper scopes, then it would make sense to also enhance typeset
to be able to do the same. Indeed, in my proof of concept, define-var
defines a new variable in the enclosing scope no matter what. In that case,
relying on a named reference is a little overkill. Using a plain typeset
would be cleaner.

Philippe

On Wed, May 7, 2025 at 1:18 PM James Widman <[email protected]> wrote:

> Hi all,
>
> I recently hit a bug in zsh 5.9 that has since been fixed in the main git
> branch, so I thought I'd describe the bug here in case anyone else runs
> into
> it.
>
> The example is:
>
> #!/usr/bin/env zsh
> foo(){
>     setopt err_return
>     source <(echo 'if true; then return 42; fi')
>     echo "This line is printed by zsh 5.9 (but it shouldn't be)."
> }
> foo
>
> I ran git-bisect and found that this bug was fixed in:
>
> commit f253ea6b9d14901e24fa6a376936effad9e6547b
> Author: Philippe Altherr [email redacted]
> Date:   Sat Dec 3 21:14:26 2022 -0800
>
>     51076: fix ERR_EXIT when used with "eval" or "source"; documentary
> comments
>
> In the scripts i was originally working on, the source'd file is named
> 'foo.parse-arguments', which parses arguments that the user provides to
> `foo`,
> and which contains an invocation of `zparseopts` (which is about a
> half-page
> long), followed by several checks to make sure that the arguments are
> reasonable.
>
> I `source` it so that the local parameters defined in `foo.parse-arguments`
> are introduced into the scope of `foo`. And because `foo.parse-arguments`
> returns non-zero in the event of a usage error, everything in `foo` after
> the
> `source` invocation can just use those option parameters without any other
> checks (and that helps to keep `foo` reasonably-sized).
>
> ...But that all depended on ERR_RETURN working as in the example above. So
> I'm
> very happy to find that this bug was already fixed!
>
> Many thanks to Philippe Altherr for all the ERR_RETURN/ERR_EXIT fixes, and
> also, thanks to all maintainers!
>
> I've been using zsh for at least 15 years. I don't recall hitting any other
> zsh bugs in that time, and I cannot say the same of most of the tools that
> I
> use on a daily basis. Well done, all!
>
> --James
>