Re: Quoting special characters (was: Re: Possible solution for special characters in makefile paths)

On Tue, 2014-02-25 at 17:09 +0100, Frank Heckenbach wrote:
> > > In the command-line it would also substituted as is, so no change from
> > > the existing behaviour here.
> > 
> > Ouch; this statement is not precise and this is THE CRUX of the problem.
> > What exactly do you mean by "substituted as is"?  As is what?
> 
> As it is stored internally, i.e. with the backslash.

Just to be clear I don't want to say that this is how things will be
stored internally.  Maybe it will be, maybe not.  Eli has a great point
that we don't need to bind these things together too tightly during the
design discussion and the most important part is the user interface.
However, I understand that what you mean is "as it was written in the
makefile".

> Yes. Actually, I think/fear make will not only have to deal with
> backslashes in word-splitting contexts, but with the full glory of
> quoting as supported by POSIX sh, including single and double
> quotes, so $^ will actually mean the same to the shell than it did
> to make in the dependency list.

Why do you think this will be needed?

> > > That's probably not what was intended, but I'd blame this on the
> > > users.
> > 
> > Well, I think it's a little harsh to blame the users for doing what 40
> > years of make history and portability have taught them is best-practice.
> 
> Is this so? I've seen a lot of makefiles that don't quote $@. (Then
> again, I've seen a lot of shell scripts that don't quote their
> variables when they really should.) So probably the best we can say
> it that there's a lot of mess out there. There's no hope to make all
> of it work correctly, so we can only choose one variant we want to
> make work.

I'm not saying that we won't have to change something.  I'm just saying
it's not fair to blame the user for not writing their makefile
"correctly".  We have to own up to the fact that we're introducing a
backward-incompatibility and hope people feel that the increased utility
is worth the pain of modifying their (previously perfectly legitimate)
makefiles.

On Tue, 2014-02-25 at 19:13 +0200, Eli Zaretskii wrote:
> > That's what I'm doing: assuming we implement your suggestion of
> > backslash-quoted whitespace, which I do agree could work _on the parsing
> > side_, let's discuss the problems that will have to be solved on the
> > expansion side as a result of this syntax.
> 
> Sorry, I didn't understand we were past that first issue.  Until very
> recently there was a lot of talk about how the proposed new meaning of
> a backslash might break too many Makefiles out there.

Yes... I was worried it might break makefiles _because of the way
expansion would be handled_.  Not because because of the way we would
parse the values--that's why I was writing lots of rules that echoed
various expansion scenarios.  Sorry for not making that clear.

> > My original proposal was that we would quote content on the input side
> > using a new quoting method.  One possible syntax I proposed was using
> > $[...] where everything inside the brackets was quoted.
> 
> OK, but the issue of how to expand $[foo bar] in various contexts
> still needs to be discussed, because (unlike with the backslash) we
> cannot leave it in the expansion.

My original proposal was a complete solution, and so it covered this.
To sum it up (you can go check the original email for a more in-depth
discussion):

On parsing $[...] (wherever it appears) we would encode the string "..."
so that any special characters like whitespace, etc. are encoded into
values that weren't special to make.  I proposed using bytecodes like
001, 002, etc. but there are other options.  This is very like the
$(quote ) etc. function that has been proposed, except with a shorter
syntax.

The encoding of the string internally allows all current make functions
to go unmodified, because they're all searching for whitespace (for
example) and those characters wouldn't be whitespace anymore.  As you
point out, the encoding is an implementation detail so not really
relevant to the UI.

We would proceed without change from today's behavior all the way
through to the very last step right before we actually invoked the
command to run.  In the slow path that means we've built up a string,
expanded all variables, etc. etc.  Right then, the very last thing we do
before we pass the string to system() (or whatever on make ports), we
decode that string and turn the special characters back into whitespace
or whatever they were before.

If we took the fast path, then after we'd broken up the string into argv
and just before we called fork/exec (or whatever on the make ports), we
would decode each value of argv individually.

Ditto for setting the environment.

There would be no attempt to add any shell quoting around special
characters into the recipe, that would be up to the user.

That's all there was to it.