Re: Bug report for sed 4.2.2: '?' not working in basic regexps?

"Daniel Goldman [email protected] [sed-users]" <[email protected]> Sun, 15 Sep 2019 19:19:37 -0700
Newsgroups gmane.editors.sed.user
Message-ID <[email protected]>
--BpgrDwwGKv6b8xTSLCNhkp5EJgCsrmq1bXtpTp5
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit

 >> `\?` is an extended-regexp feature, you have to turn it on.

Admittedly, this is a tricky subject. But I do not think \? needs to be 
"turned on". From section 5.2 of the gnu sed manual:

== quote on
In GNU sed, the only difference between basic and extended regular 
expressions is in the behavior of a few special characters: ‘?’, ‘+’, 
parentheses, braces (‘{}’), and ‘|’. With basic (BRE) syntax, these 
characters do not have special meaning unless prefixed with a backslash 
(‘\’); While with extended (ERE) syntax it is reversed: these characters 
are special unless they are prefixed with backslash (‘\’).
== quote off

So it clearly says that ? (zero or one of previous character) DOES have 
meaning in BRE if prefixed with a backslash. So you do not have to “turn 
it on”. The -E flag (I use -r flag for some reason) simply switches the 
meaning:

? is literal in BRE, special meaning in ERE
\? is special meaning in BRE, literal in ERE

Related to Jim's example, it behaves exactly as expected, interpreting 
\? as literal question mark since -E flag used:

$ echo 'one?two' | sed -E 's/e\?//' # works
ontwo

$ echo 'one?two' | sed "s/e?//" # same thing
ontwo

I think we are missing something. One problem is that the shell 
interprets backslash in different ways depending on whether quotes are 
used (or not) and what kind of quotes. With single quotes, backslash has 
no special meaning, is not interpreted by the shell, and is passed to 
sed. With double quotes, backslash MAY have a special meaning, may be 
interpreted by the shell before passing the contents to sed, if 
backslash is followed by backslash, dollar sign, back quote, newline or 
inner double quote. If no quotes, backslash is interpreted by shell, 
quotes next character.

Anyway, the following examples show \? working even if not “turned on”:

$ echo THAT | sed "s/T\?HAT\?/++/" # works
++
$ echo HA | sed "s/T\?HAT\?/++/" # works
++
$ echo WHAT | sed "s/T\?HAT\?/++/" # works
W++
$ echo THAW | sed "s/T\?HAT\?/++/" # works
++W

To me the gnu sed manual seems correct, since it matches the above 
behavior. However, the poster’s example with the e\? regular expression 
does seem more difficult to figure out. So it is an excellent post, IMO. 
It is possible sed is not working correctly, but I’m not ready to say 
that yet. I would need to look more when I have some time.

I would point out the following interesting (to me) results, almost like 
\? is usually greedy (as it should be), but maybe NOT greedy with e\? 
(or similar) regular expression:

$ echo 'one?two' | sed "s/ne\?//" # works
o?two
$ echo 'one?two' | sed "s/e\?.t//" # works
onwo
$ echo 'one?two' | sed "s/.\?//" # works
ne?two
$ echo 'one?two' | sed "s/e\?//" # seems to fail
one?two
$ echo 'one?two' | sed -E "s/e?//" # seems to fail
one?two

Anybody else?

Daniel

On 9/15/2019 1:43 PM, Jim Hill [email protected] [sed-users] wrote:
> echo 'one?two' | sed -E 's/e\?//'

--BpgrDwwGKv6b8xTSLCNhkp5EJgCsrmq1bXtpTp5
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable




<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/htm=
l4/strict.dtd">
<html>
<head>
</head>






=20
<body style=3D"background-color: #fff;">
<span style=3D"display:none">&nbsp;</span>

<!--~-|**|PrettyHtmlStartT|**|-~-->
<div id=3D"ygrp-mlmsg" style=3D"position:relative;">
  <div id=3D"ygrp-msg" style=3D"z-index: 1;">
<!--~-|**|PrettyHtmlEndT|**|-~-->

    <div id=3D"ygrp-text" >
=20=20=20=20=20=20
=20=20=20=20=20=20
      <p> &gt;&gt; `&#92;?` is an extended-regexp feature, you have to turn=
 it on.<br>
<br>
Admittedly, this is a tricky subject. But I do not think &#92;? needs to be=
 <br>
&quot;turned on&quot;. From section 5.2 of the gnu sed manual:<br>
<br>
=3D=3D quote on<br>
In GNU sed, the only difference between basic and extended regular <br>
expressions is in the behavior of a few special characters: =E2=80=98?=E2=
=80=99, =E2=80=98+=E2=80=99, <br>
parentheses, braces (=E2=80=98{}=E2=80=99), and =E2=80=98|=E2=80=99. With b=
asic (BRE) syntax, these <br>
characters do not have special meaning unless prefixed with a backslash <br=
>
(=E2=80=98&#92;=E2=80=99); While with extended (ERE) syntax it is reversed:=
 these characters <br>
are special unless they are prefixed with backslash (=E2=80=98&#92;=E2=80=
=99).<br>
=3D=3D quote off<br>
<br>
So it clearly says that ? (zero or one of previous character) DOES have <br=
>
meaning in BRE if prefixed with a backslash. So you do not have to =E2=80=
=9Cturn <br>
it on=E2=80=9D. The -E flag (I use -r flag for some reason) simply switches=
 the <br>
meaning:<br>
<br>
? is literal in BRE, special meaning in ERE<br>
&#92;? is special meaning in BRE, literal in ERE<br>
<br>
Related to Jim&#39;s example, it behaves exactly as expected, interpreting =
<br>
&#92;? as literal question mark since -E flag used:<br>
<br>
$ echo &#39;one?two&#39; | sed -E &#39;s/e&#92;?//&#39; # works<br>
ontwo<br>
<br>
$ echo &#39;one?two&#39; | sed &quot;s/e?//&quot; # same thing<br>
ontwo<br>
<br>
I think we are missing something. One problem is that the shell <br>
interprets backslash in different ways depending on whether quotes are <br>
used (or not) and what kind of quotes. With single quotes, backslash has <b=
r>
no special meaning, is not interpreted by the shell, and is passed to <br>
sed. With double quotes, backslash MAY have a special meaning, may be <br>
interpreted by the shell before passing the contents to sed, if <br>
backslash is followed by backslash, dollar sign, back quote, newline or <br=
>
inner double quote. If no quotes, backslash is interpreted by shell, <br>
quotes next character.<br>
<br>
Anyway, the following examples show &#92;? working even if not =E2=80=9Ctur=
ned on=E2=80=9D:<br>
<br>
$ echo THAT | sed &quot;s/T&#92;?HAT&#92;?/&#43;+/&quot; # works<br>
&#43;+<br>
$ echo HA | sed &quot;s/T&#92;?HAT&#92;?/&#43;+/&quot; # works<br>
&#43;+<br>
$ echo WHAT | sed &quot;s/T&#92;?HAT&#92;?/&#43;+/&quot; # works<br>
W&#43;+<br>
$ echo THAW | sed &quot;s/T&#92;?HAT&#92;?/&#43;+/&quot; # works<br>
&#43;+W<br>
<br>
To me the gnu sed manual seems correct, since it matches the above <br>
behavior. However, the poster=E2=80=99s example with the e&#92;? regular ex=
pression <br>
does seem more difficult to figure out. So it is an excellent post, IMO. <b=
r>
It is possible sed is not working correctly, but I=E2=80=99m not ready to s=
ay <br>
that yet. I would need to look more when I have some time.<br>
<br>
I would point out the following interesting (to me) results, almost like <b=
r>
&#92;? is usually greedy (as it should be), but maybe NOT greedy with e&#92=
;? <br>
(or similar) regular expression:<br>
<br>
$ echo &#39;one?two&#39; | sed &quot;s/ne&#92;?//&quot; # works<br>
o?two<br>
$ echo &#39;one?two&#39; | sed &quot;s/e&#92;?.t//&quot; # works<br>
onwo<br>
$ echo &#39;one?two&#39; | sed &quot;s/.&#92;?//&quot; # works<br>
ne?two<br>
$ echo &#39;one?two&#39; | sed &quot;s/e&#92;?//&quot; # seems to fail<br>
one?two<br>
$ echo &#39;one?two&#39; | sed -E &quot;s/e?//&quot; # seems to fail<br>
one?two<br>
<br>
Anybody else?<br>
<br>
Daniel<br>
<br>
On 9/15/2019 1:43 PM, Jim Hill [email protected] [sed-users] wrote:<br>
&gt; echo &#39;one?two&#39; | sed -E &#39;s/e&#92;?//&#39;<br>
</p>

    </div>
=20=20=20=20=20

    <!--~-|**|PrettyHtmlStart|**|-~-->
    <div style=3D"color: #fff; height: 0;">__._,_.___</div>

=20=20=20=20=20=20=20=20=20=20
=20=20
=20

=20=20=20=20
    <div style=3D"clear:both"> </div>

    <div id=3D"fromDMARC" style=3D"margin-top: 10px;">
        <hr style=3D"height:2px ; border-width:0; color:#E3E3E3; background=
-color:#E3E3E3;">
        Posted by: Daniel Goldman &lt;[email protected]&gt;        <hr styl=
e=3D"height:2px ; border-width:0; color:#E3E3E3; background-color:#E3E3E3;"=
>
     </div>
    <div style=3D"clear:both"> </div>

    <table cellspacing=3D4px style=3D"margin-top: 10px; margin-bottom: 10px=
; color: #2D50FD;">
      <tbody>
        <tr>
          <td style=3D"font-size: 12px; font-family: arial; font-weight: bo=
ld; padding: 7px 5px 5px;"  >
                          <a style=3D"text-decoration: none; color: #2D50FD=
" href=3D"https://groups.yahoo.com/neo/groups/sed-users/conversations/messa=
ges/9995;_ylc=3DX3oDMTJwM284aXZiBF9TAzk3MzU5NzE0BGdycElkAzI0ODk2MzkEZ3Jwc3B=
JZAMxNzA5MzM1MDAyBG1zZ0lkAzk5OTUEc2VjA2Z0cgRzbGsDcnBseQRzdGltZQMxNTY4NjAwND=
A4?act=3Dreply&messageNum=3D9995">Reply via web post</a>
                      </td>
          <td>&bull;</td>
          <td style=3D"font-size: 12px; font-family: arial; padding: 7px 5p=
x 5px;" >
            <a href=3D"mailto:[email protected]?subject=3DRe%3A%20Bug%20rep=
ort%20for%20sed%204%2E2%2E2%3A%20%27%3F%27%20not%20working%20in%20basic%20r=
egexps%3F" style=3D"text-decoration: none; color: #2D50FD;">
               Reply to sender            </a>
          </td>
          <td>&bull;</td>
          <td style=3D"font-size: 12px; font-family: arial; padding: 7px 5p=
x 5px;">
            <a href=3D"mailto:[email protected]?subject=3DRe%3A%20B=
ug%20report%20for%20sed%204%2E2%2E2%3A%20%27%3F%27%20not%20working%20in%20b=
asic%20regexps%3F" style=3D"text-decoration: none; color: #2D50FD">
              Reply to group            </a>
          </td>
          <td>&bull;</td>
          <td style=3D"font-size: 12px; font-family: arial; padding: 7px 5p=
x 5px;" >
            <a href=3D"https://groups.yahoo.com/neo/groups/sed-users/conver=
sations/newtopic;_ylc=3DX3oDMTJlNTBoMXJ2BF9TAzk3MzU5NzE0BGdycElkAzI0ODk2Mzk=
EZ3Jwc3BJZAMxNzA5MzM1MDAyBHNlYwNmdHIEc2xrA250cGMEc3RpbWUDMTU2ODYwMDQwOA--" =
style=3D"text-decoration: none; color: #2D50FD">Start a New Topic</a>
          </td>
          <td>&bull;</td>
          <td style=3D"font-size: 12px; font-family: arial; padding: 7px 5p=
x 5px;color: #2D50FD;" >
                            <a href=3D"https://groups.yahoo.com/neo/groups/=
sed-users/conversations/topics/9993;_ylc=3DX3oDMTM0Z2lwcHBvBF9TAzk3MzU5NzE0=
BGdycElkAzI0ODk2MzkEZ3Jwc3BJZAMxNzA5MzM1MDAyBG1zZ0lkAzk5OTUEc2VjA2Z0cgRzbGs=
DdnRwYwRzdGltZQMxNTY4NjAwNDA4BHRwY0lkAzk5OTM-" style=3D"text-decoration: no=
ne; color: #2D50FD;">Messages in this topic</a>
                (3)
                      </td>
        </tr>
      </tbody>
    </table>

=20=20=20=20=20=20=20=20

<!------- Start Nav Bar ------>


    <div id=3D"ygrp-grfd" style=3D"font-family: Verdana; font-size: 12px; p=
adding: 15px 0;">
=20=20=20=20=20=20
<!-- |**|begin egp html banner|**| -->

      -- <BR>
=20=20=20=20=20=20
<!-- |**|end egp html banner|**| -->

    </div>
=20=20

=20

<!-- |**|begin egp html banner|**| -->
<div id=3D"ygrp-vital" style=3D"background-color: #f2f2f2; font-family: Ver=
dana; font-size: 10px; margin-bottom: 10px; padding: 10px;">

    <span id=3D"vithd" style=3D"font-weight: bold; color: #333; text-transf=
orm: uppercase; "><a href=3D"https://groups.yahoo.com/neo/groups/sed-users/=
info;_ylc=3DX3oDMTJlcW5odHNkBF9TAzk3MzU5NzE0BGdycElkAzI0ODk2MzkEZ3Jwc3BJZAM=
xNzA5MzM1MDAyBHNlYwN2dGwEc2xrA3ZnaHAEc3RpbWUDMTU2ODYwMDQwOA--" style=3D"tex=
t-decoration: none;">Visit Your Group</a></span>

     <ul style=3D"list-style-type: none; margin: 0; padding: 0; display: in=
line;">
                                                    </ul>
  </div>


<div id=3D"ft" style=3D"font-family: Arial; font-size: 11px; margin-top: 5p=
x; padding: 0 2px 0 0; clear: both;">
  <a href=3D"https://groups.yahoo.com/neo;_ylc=3DX3oDMTJkOTVpdms4BF9TAzk3Mz=
U5NzE0BGdycElkAzI0ODk2MzkEZ3Jwc3BJZAMxNzA5MzM1MDAyBHNlYwNmdHIEc2xrA2dmcARzd=
GltZQMxNTY4NjAwNDA4" style=3D"float: left;"><img src=3D"http://l.yimg.com/r=
u/static/images/yg/img/email/new_logo/logo-groups-137x15.png" height=3D"15"=
 width=3D"137" alt=3D"Yahoo! Groups" style=3D"border: 0;"/></a>
  <div style=3D"color: #747575; float: right;"> &bull; <a href=3D"https://i=
nfo.yahoo.com/privacy/us/yahoo/groups/details.html" style=3D"text-decoratio=
n: none;">Privacy</a> &bull; <a href=3D"mailto:sed-users-unsubscribe@yahoog=
roups.com?subject=3DUnsubscribe" style=3D"text-decoration: none;">Unsubscri=
be</a> &bull; <a href=3D"https://info.yahoo.com/legal/us/yahoo/utos/terms/"=
 style=3D"text-decoration: none;">Terms of Use</a> </div>
</div>
<br>

<!-- |**|end egp html banner|**| -->

  </div> <!-- ygrp-msg -->

=20
  <!-- Sponsor -->
  <!-- |**|begin egp html banner|**| -->
  <div id=3D"ygrp-sponsor" style=3D"width:160px; float:right; clear:none; m=
argin:0 0 25px 0; background: #fff;">

<!-- Start Recommendations -->
<div id=3D"ygrp-reco">
     </div>
<!-- End Recommendations -->



  </div>   <!-- |**|end egp html banner|**| -->

  <div style=3D"clear:both; color: #FFF; font-size:1px;">.</div>
</div>

  <img src=3D"http://geo.yahoo.com/serv?s=3D97359714/grpId=3D2489639/grpspI=
d=3D1709335002/msgId=3D9995/stime=3D1568600408" width=3D"1" height=3D"1"> <=
br>

<img src=3D"http://y.analytics.yahoo.com/fpc.pl?ywarid=3D515FB27823A7407E&a=
=3D10001310322279&js=3Dno&resp=3Dimg" width=3D"1" height=3D"1">=20

<div style=3D"color: #fff; height: 0;">__,_._,___</div>
<!--~-|**|PrettyHtmlEnd|**|-~-->

</body>

<!--~-|**|PrettyHtmlStart|**|-~-->
<head>
  <style type=3D"text/css">
  <!--
  #ygrp-mkp {
  border: 1px solid #d8d8d8;
  font-family: Arial;
  margin: 10px 0;
  padding: 0 10px;
}

#ygrp-mkp hr {
  border: 1px solid #d8d8d8;
}

#ygrp-mkp #hd {
  color: #628c2a;
  font-size: 85%;
  font-weight: 700;
  line-height: 122%;
  margin: 10px 0;
}

#ygrp-mkp #ads {
  margin-bottom: 10px;
}

#ygrp-mkp .ad {
  padding: 0 0;
}

#ygrp-mkp .ad p {
  margin: 0;
}

#ygrp-mkp .ad a {
  color: #0000ff;
  text-decoration: none;
}
  #ygrp-sponsor #ygrp-lc {
  font-family: Arial;
}

#ygrp-sponsor #ygrp-lc #hd {
  margin: 10px 0px;
  font-weight: 700;
  font-size: 78%;
  line-height: 122%;
}

#ygrp-sponsor #ygrp-lc .ad {
  margin-bottom: 10px;
  padding: 0 0;
}

  #actions {
    font-family: Verdana;
    font-size: 11px;
    padding: 10px 0;
  }

  #activity {
    background-color: #e0ecee;
    float: left;
    font-family: Verdana;
    font-size: 10px;
    padding: 10px;
  }

  #activity span {
    font-weight: 700;
  }

  #activity span:first-child {
    text-transform: uppercase;
  }

  #activity span a {
    color: #5085b6;
    text-decoration: none;
  }

  #activity span span {
    color: #ff7900;
  }

  #activity span .underline {
    text-decoration: underline;
  }

  .attach {
    clear: both;
    display: table;
    font-family: Arial;
    font-size: 12px;
    padding: 10px 0;
    width: 400px;
  }

  .attach div a {
    text-decoration: none;
  }

  .attach img {
    border: none;
    padding-right: 5px;
  }

  .attach label {
    display: block;
    margin-bottom: 5px;
  }

  .attach label a {
    text-decoration: none;
  }
=20=20
  blockquote {
    margin: 0 0 0 4px;
  }

  .bold {
    font-family: Arial;
    font-size: 13px;
    font-weight: 700;
  }

  .bold a {
    text-decoration: none;
  }

  dd.last p a {
    font-family: Verdana;
    font-weight: 700;
  }

  dd.last p span {
    margin-right: 10px;
    font-family: Verdana;
    font-weight: 700;
  }

  dd.last p span.yshortcuts {
    margin-right: 0;
  }

  div.attach-table div div a {
    text-decoration: none;
  }

  div.attach-table {
    width: 400px;
  }

  div.file-title a, div.file-title a:active, div.file-title a:hover, div.fi=
le-title a:visited {
    text-decoration: none;
  }

  div.photo-title a, div.photo-title a:active, div.photo-title a:hover, div=
..photo-title a:visited {
    text-decoration: none;
  }

  div#ygrp-mlmsg #ygrp-msg p a span.yshortcuts {
    font-family: Verdana;
    font-size: 10px;
    font-weight: normal;
  }

  .green {
    color: #628c2a;
  }

  .MsoNormal {
    margin: 0 0 0 0;
  }

  o {
    font-size: 0;
  }

  #photos div {
    float: left;
    width: 72px;
  }

  #photos div div {
    border: 1px solid #666666;
    height: 62px;
    overflow: hidden;
    width: 62px;
  }

  #photos div label {
    color: #666666;
    font-size: 10px;
    overflow: hidden;
    text-align: center;
    white-space: nowrap;
    width: 64px;
  }

  #reco-category {
    font-size: 77%;
  }

  #reco-desc {
    font-size: 77%;
  }

  .replbq {
    margin: 4px;
  }

  #ygrp-actbar div a:first-child {
   /* border-right: 0px solid #000;*/
    margin-right: 2px;
    padding-right: 5px;
  }

  #ygrp-mlmsg {
    font-size: 13px;
    font-family: Arial, helvetica,clean, sans-serif;
    *font-size: small;
    *font: x-small;
  }

  #ygrp-mlmsg table {
    font-size: inherit;
    font: 100%;
  }

  #ygrp-mlmsg select, input, textarea {
    font: 99% Arial, Helvetica, clean, sans-serif;
  }

  #ygrp-mlmsg pre, code {
    font:115% monospace;
    *font-size:100%;
  }

  #ygrp-mlmsg * {
    line-height: 1.22em;
  }

  #ygrp-mlmsg #logo {
    padding-bottom: 10px;
  }


  #ygrp-msg p a {
    font-family: Verdana;
  }

  #ygrp-msg p#attach-count span {
    color: #1E66AE;
    font-weight: 700;
  }

  #ygrp-reco #reco-head {
    color: #ff7900;
    font-weight: 700;
  }

  #ygrp-reco {
    margin-bottom: 20px;
    padding: 0px;
  }

  #ygrp-sponsor #ov li a {
    font-size: 130%;
    text-decoration: none;
  }

  #ygrp-sponsor #ov li {
    font-size: 77%;
    list-style-type: square;
    padding: 6px 0;
  }=20

  #ygrp-sponsor #ov ul {
    margin: 0;
    padding: 0 0 0 8px;
  }

  #ygrp-text {
    font-family: Georgia;
  }

  #ygrp-text p {
    margin: 0 0 1em 0;
  }

  #ygrp-text tt {
    font-size: 120%;
  }

  #ygrp-vital ul li:last-child {
    border-right: none !important;=20
  }=20
  -->
  </style>
</head>

<!--~-|**|PrettyHtmlEnd|**|-~-->
</html>
<!-- end group email -->


--BpgrDwwGKv6b8xTSLCNhkp5EJgCsrmq1bXtpTp5--