Re: Is it fair to say, "sed supports # comments starting any column"?

2015-08-11 22:37:42 -0700, Daniel Goldman [email protected] [sed-users]:
> Hi Stephane,
> 
> Thanks for your thoughtful reply.
> 
> - The POSIX spec, at the link I sent, does not say "# is a command".


The POSIX spec for sed is a total mess IMO. It leaves plenty of
things unspecified. It specifies idiosyncraties of early
implementations instead of leaving the behaviour unspecified
(like !! is to be the same as !), it requires things that no sed
implementations ever required. 


However it does say "#" is a command verb and is listed among
the other command verbs.


It does say the semicolon can be used to separate commands and
it does say blanks are allowed around addresses.




> The POSIX text is confusingly worded. What else is new? Even the GNU sed 
> manual sloppily lumps in # with other "commands".


Except for the b/t/: commands, it's allowed to do that as it's
unspecified in POSIX.


> But a comment is not a 
> command. It's a comment. A command must be separated from other 
> commands. Neither POSIX nor the GNU sed manual say anything about having 
> to separate # from other commands on the same line. But I am open to 
> other information anyone might provide.


It is a command in the POSIX spec. GNU sed doesn't treat it as a
command. It's OK to do that as long as it's not otherwise
breaking conformance.


g # foo


is OK. g doesn't take arguments, so if you write "g # foo" in
your script, your script is not a POSIX conformant script, so
returning an error or ignoring that "# foo" are two conforming
behaviours as it's not specified.


> - The POSIX spec, at the link I sent, does not say "it (#) must be 
> separated from another command with ; or newline".


It does say a command other than a few can be followed by
semicolon followed by another command (# being no exception).


> I think the standard allows "s/x/y/ # comment" format.


That is left unspecified. If you do "s/x/y/ # comment", you have
no guarantee of the outcome, that " # comment" could be ignored,
or you can get an error, or "#" could be treated as a s flag as
an extension.


> 
> - The POSIX spec, at the link I sent, does not mention ; syntax. If 
> anyone has a POSIX link that mentions ; syntax, please post it. I'm not 
> saying you are wrong. I would very much hope that POSIX would mention 
> the very useful ; syntax! If it does not, could it please by added? I 
> just don't see the mention, so the point about "s/x/y/; # comment" being 
> required does not make sense.


Read the spec you posted again, and search for "semicolon"


> For some of the examples you post, probably best for me not to respond 
> unless someone can post the apparently different POSIX citation that 
> backs up the wordings you quote. You obviously have expertise with POSIX 
> specs. I hope you can provide a better link.
> 
> Could you clarify what you concluded about the solaris sed versions? I'm 
> not sure what "command -p" does.


I conclude that Solaris is not compliant. "command -p cmd" is
meant to invoke the POSIX (portable) version of cmd. In the case
of Solaris, that's /usr/xpg4/bin/sed.


[...]
> Is my understanding correct, based on "SVID which only allows a comment 
> as the first line of the script", and what Rakesh said, that System V 
> only allows the line 1 of the script to be a comment? If so, isn't that 
> incredibly pitiful, and no basis for any kind of argument? It seems 
> extremely primitive, which is saying a lot coming from a seder. :)
[...]


That was specifying the behaviour at the time (80s). It was
saying that you shouldn't write sed ' # comment' because SysV
sed implementations didn't support it. It was not preventing
conforming sed implementations from supporting it.


I don't expect any modern system would follow that standard (at
least not on the aspect where it's not compatible with POSIX (if
any)).


> AFAIK, the examples with the w command have nothing to do with this 
> comment issue, as exactly the same problem occurs with other sed syntax. 
> In other words, the w command has to end a script line, regardless of 
> what follows. Even ; after the w command will cause an error (assuming 
> the file name does not contain ; character).


It's relevant is that when you ask: "can a comment start in any
column?", the answer is no because in:


#! /usr/bin/sed -f
w some # long\
# file with newline


"#" doesn't start a comment but is considered as part of the
file name. In my understanding, that's meant to write to the
"some # long<newline># file with newline" file (so GNU sed is
not compliant here as it writes to "some # long\" instead,
though that's not a big issue as there's little chance anyone
would want to do that).


> You are right that # is not an s flag. It never will be, that would 
> break backward compatibility


That would not break POSIX sed scripts. That may break some
scripts that rely on the specific behaviour of GNU sed to accept
comments anywhere.




> choice. Of course, sed 's/x/y/#comment' is asking for trouble, a kind of 
> "coding horror". But it is perfectly legal, and in accord with the POSIX 
> standard, to my understanding.


No, AFAICT, there's nothing in the POSIX spec that specifies
what the effect of s/x/y/#comment is.


> And from your answer, it sounds like 
> there is no other standard that overrides POSIX.


There are standards that systems may want to follow that
override or complement POSIX (like the Debian policy, the Linux
Standard Base, options XSI (Unix) in the Single Unix
Specification beyond the base). I meant that I was not aware of
a standard that would disagree with POSIX on this.


> The GNU sed online manual does not list "comments can start in any 
> column" as an GNU extension. Apparently, the maintainers consider GNU 
> sed # behavior to be POSIX compliant. To me, combined with the POSIX 
> wording I have seen so far, that indicates the answer to my question, 
> "is it fair to say sed supports # comments starting any column?" is 
> "YES". Would anyone argue "NO", and on what basis?


No, as said already


w blah # blah


doesn't start a comment even in GNU sed.


> To my knowledge, 99% to 100% of languages do not require a command 
> separator before a comment. To me, the idea is ridiculous. The POSIX 
> standard is confusingly worded. But I doubt 100% that the POSIX 
> designers intended to require "s/x/y/; # comment". What would be the 
> point?


POSIX is a standard to help write portable applications. In the
early days at least, it was descriptive (it's now becoming more
prescriptive). The specification was specifying what you could
and could not do for your script to be portable to the different
compliant implementations around. And that's how sed (which
predates most other languages that have # as a comment) was
implemented.


> I've used sed for decades, and the idea of "s/x/y/; # comment" is 
> laughable, even if POSIX mentioned ; syntax.


It may be laughable, but it will break with many (most) sed
implementations including the ones directly derived from the
original implementation and others that don't like FreeBSD's if
you use s/x/y/ # comment.


> As you point out, "fixing" 
> a sed version to allow "s/x/y/ # comment" would not break existing 
> scripts. In contrast, requiring "s/x/y/; # comment" syntax would break 
> scripts. :( Thus, my answer to "must sed # comments be separated from 
> commands on the same line by ; syntax?" is "NO". Would anyone argue 
> "YES", and on what basis?


YES, because many sed implementations will fail otherwise and
they are allowed to by POSIX.


The joy of writing portable script is having to deal with all
the idiosyncraties of the various implementations. POSIX is a
great tool to tell you what you may or may not do. In the case
of sed though, I'd agree the POSIX spec is "sub-standard" (pun
intended).


-- 
Stephane