Re: [perl #133541] Grammer bug <alnum> vs <alpha>
[email protected] (Brent Laabs) Fri, 28 Sep 2018 14:54:04 -0700
| Newsgroups | perl.perl6.compiler |
|---|---|
| Message-ID | <CAAPDpByj2MHC8ew-HhhB1=1=3sz2cr1+2Ua-Nf6Yu_7AZEAoyQ@mail.gmail.com> |
--0000000000004a40800576f57f12
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Golfs to just the top grammar, which is the only one that returns Nil.
grammar Alnum1 {
token TOP {<alnum>|<alnum>.*}
}
grammar AlnumReversed {
token TOP {<alnum>.*|<alnum>}
}
grammar Alpha1 {
token TOP {<alpha>|<alpha>.*}
}
my $rx =3D rx/^ [<alnum>|<alnum>.*] $/;
my $str=3D"n~";
.say for "=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D <alnum> =3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D",
Alnum1.parse($str),
"=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D <alnum> (reversed) =3D=3D=3D",
AlnumReversed.parse($str),
"=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D <alpha> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D",
Alpha1.parse($str),
"=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Regex =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D",
$str ~~ $rx;
On Fri, Sep 28, 2018 at 7:19 AM Patrick R. Michaud via RT <
[email protected]> wrote:
> The issue doesn't seem to be the underscore, because I get the same resul=
t
> even when converting the underscore into a letter ('b'):
>
> $ cat gentb.p6
> grammar G0 {
> token TOP {<rport>|<ruport>.*}
> regex rport { <type>}
> rule ruport { <type>}
> #token type {<alpha>+}
> token type {<alnum>+}
> }
>
> grammar G1 {
> token TOP {<rport>|<ruport>.*}
> regex rport { <type>}
> rule ruport { <type>}
> token type {<alpha>+}
> #token type {<alnum>+}
> }
> my $str=3D"scbin<foo> bar";
> say "=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D <alnum> Example=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D";
> say G0.parse($str);
> say "=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D <alpha> Example=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D";
> say G1.parse($str);
>
> $ perl6 gentb.p6
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D <alnum> Example=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D
> Nil
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D <alpha> Example=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D
> =EF=BD=A2scbin<foo> bar=EF=BD=A3
> ruport =3D> =EF=BD=A2scbin=EF=BD=A3
> type =3D> =EF=BD=A2scbin=EF=BD=A3
> alpha =3D> =EF=BD=A2s=EF=BD=A3
> alpha =3D> =EF=BD=A2c=EF=BD=A3
> alpha =3D> =EF=BD=A2b=EF=BD=A3
> alpha =3D> =EF=BD=A2i=EF=BD=A3
> alpha =3D> =EF=BD=A2n=EF=BD=A3
> $
>
>
> On Fri, Sep 28, 2018 at 02:26:41AM -0700, Brent Laabs wrote:
> > Are you sure about that? Underscore has been part of the specs
> (synopses)
> > for <alpha> for at least 10 years, probably longer.
> >
> > > "_" ~~ /<alpha>/
> > =EF=BD=A2_=EF=BD=A3
> > alpha =3D> =EF=BD=A2_=EF=BD=A3
> >
> > On Thu, Sep 27, 2018 at 7:52 PM Brandon Allbery <[email protected]>
> wrote:
> >
> > > "_" is not an alphabetic character. It's allowed in "alnum" because
> that
> > > is by intent what is \w in other regex implementations, which include=
s
> "_".
> > >
> > > On Thu, Sep 27, 2018 at 10:47 PM Vijayvithal <
> [email protected]>
> > > wrote:
> > >
> > >> # New Ticket Created by Vijayvithal
> > >> # Please include the string: [perl #133541]
> > >> # in the subject line of all future correspondence about this issue.
> > >> # <URL: https://rt.perl.org/Ticket/Display.html?id=3D133541 >
> > >>
> > >>
> > >> In the attached code, the only difference between the Grammars G0 an=
d
> G1
> > >> is the defination of token 'type' it is defined as <alpha> in one ca=
se
> > >> and as <alnum> in another.
> > >>
> > >> Since the string being matched is 'sc_in' both the alpha and alnum
> > >> tokens should have captured it. But we see the following result on
> > >> execution
> > >>
> > >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D <alnum> Example=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D
> > >> Nil
> > >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D <alpha> Example=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D
> > >> =EF=BD=A2sc_in<foo> bar=EF=BD=A3
> > >> ruport =3D> =EF=BD=A2sc_in=EF=BD=A3
> > >> type =3D> =EF=BD=A2sc_in=EF=BD=A3
> > >> alpha =3D> =EF=BD=A2s=EF=BD=A3
> > >> alpha =3D> =EF=BD=A2c=EF=BD=A3
> > >> alpha =3D> =EF=BD=A2_=EF=BD=A3
> > >> alpha =3D> =EF=BD=A2i=EF=BD=A3
> > >> alpha =3D> =EF=BD=A2n=EF=BD=A3
> > >>
> > >>
> > >> Perl Version is
> > >>
> > >> This is Rakudo Star version 2018.06 built on MoarVM version 2018.06
> > >> implementing Perl 6.c.
> > >>
> > >>
> > >>
> > >> --
> > >> Vijayvithal
> > >> Dyumnin Semiconductors
> > >>
> > >
> > >
> > > --
> > > brandon s allbery kf8nh
> > > [email protected]
> > >
>
>
--0000000000004a40800576f57f12
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
<div dir=3D"ltr"><div dir=3D"ltr"><div>Golfs to just the top grammar, which=
is the only one that returns Nil.<br></div><div><br></div><div>grammar Aln=
um1 {<br>=C2=A0=C2=A0=C2=A0 token TOP {<alnum>|<alnum>.*}<br>}<=
br>grammar AlnumReversed {<br>=C2=A0=C2=A0=C2=A0 token TOP {<alnum>.*=
|<alnum>}<br>}<br>grammar Alpha1 {<br>=C2=A0=C2=A0=C2=A0 token TOP {&=
lt;alpha>|<alpha>.*}<br>}<br>my $rx =3D rx/^ [<alnum>|<al=
num>.*] $/;</div><div><br></div><div>my $str=3D"n~";<br><br>.s=
ay for "=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D <alnum> =3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D",<br>=C2=A0Alnum1.parse($str),<br>=C2=
=A0"=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D <alnum> (reversed) =3D=3D=
=3D",<br>=C2=A0AlnumReversed.parse($str),<br>=C2=A0"=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D <alpha> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D",<br>=C2=A0Alpha1.parse($str),<br>=C2=A0"=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D Regex=C2=A0=C2=A0 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D",<br>=C2=A0$str ~~ $rx;<br><br></div></div></div><br><div class=3D=
"gmail_quote"><div dir=3D"ltr">On Fri, Sep 28, 2018 at 7:19 AM Patrick R. M=
ichaud via RT <<a href=3D"mailto:[email protected]">perl6-bug=
[email protected]</a>> wrote:<br></div><blockquote class=3D"gmail_quot=
e" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">=
The issue doesn't seem to be the underscore, because I get the same res=
ult even when converting the underscore into a letter ('b'):<br>
<br>
$ cat gentb.p6 <br>
grammar G0 {<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 token TOP {<rport>|<ruport>.*}<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 regex rport { <type>}<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 rule ruport { <type>}<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 #token type {<alpha>+}<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 token type {<alnum>+}<br>
}<br>
<br>
grammar G1 {<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 token TOP {<rport>|<ruport>.*}<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 regex rport { <type>}<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 rule ruport { <type>}<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 token type {<alpha>+}<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 #token type {<alnum>+}<br>
}<br>
my $str=3D"scbin<foo> bar";<br>
say "=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D <alnum> Example=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D";<br>
say G0.parse($str);<br>
say "=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D <alpha> Example=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D";<br>
say G1.parse($str);<br>
<br>
$ perl6 gentb.p6<br>
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D <alnum> Example=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D<br>
Nil<br>
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D <alpha> Example=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D<br>
=EF=BD=A2scbin<foo> bar=EF=BD=A3<br>
=C2=A0ruport =3D> =EF=BD=A2scbin=EF=BD=A3<br>
=C2=A0 type =3D> =EF=BD=A2scbin=EF=BD=A3<br>
=C2=A0 =C2=A0alpha =3D> =EF=BD=A2s=EF=BD=A3<br>
=C2=A0 =C2=A0alpha =3D> =EF=BD=A2c=EF=BD=A3<br>
=C2=A0 =C2=A0alpha =3D> =EF=BD=A2b=EF=BD=A3<br>
=C2=A0 =C2=A0alpha =3D> =EF=BD=A2i=EF=BD=A3<br>
=C2=A0 =C2=A0alpha =3D> =EF=BD=A2n=EF=BD=A3<br>
$ <br>
<br>
<br>
On Fri, Sep 28, 2018 at 02:26:41AM -0700, Brent Laabs wrote:<br>
> Are you sure about that?=C2=A0 Underscore has been part of the specs (=
synopses)<br>
> for <alpha> for at least 10 years, probably longer.<br>
> <br>
>=C2=A0 >=C2=A0 "_" ~~ /<alpha>/<br>
> =EF=BD=A2_=EF=BD=A3<br>
>=C2=A0 alpha =3D> =EF=BD=A2_=EF=BD=A3<br>
> <br>
> On Thu, Sep 27, 2018 at 7:52 PM Brandon Allbery <<a href=3D"mailto:=
[email protected]" target=3D"_blank">[email protected]</a>> wrote:<b=
r>
> <br>
> > "_" is not an alphabetic character. It's allowed in=
"alnum" because that<br>
> > is by intent what is \w in other regex implementations, which inc=
ludes "_".<br>
> ><br>
> > On Thu, Sep 27, 2018 at 10:47 PM Vijayvithal <<a href=3D"mailt=
o:[email protected]" target=3D"_blank">perl6-bugs-followup@perl.=
org</a>><br>
> > wrote:<br>
> ><br>
> >> # New Ticket Created by=C2=A0 Vijayvithal<br>
> >> # Please include the string:=C2=A0 [perl #133541]<br>
> >> # in the subject line of all future correspondence about this=
issue.<br>
> >> # <URL: <a href=3D"https://rt.perl.org/Ticket/Display.html=
?id=3D133541" rel=3D"noreferrer" target=3D"_blank">https://rt.perl.org/Tick=
et/Display.html?id=3D133541</a> ><br>
> >><br>
> >><br>
> >> In the attached code, the only difference between the Grammar=
s G0 and G1<br>
> >> is the defination of token 'type' it is defined as &l=
t;alpha> in one case<br>
> >> and as <alnum> in another.<br>
> >><br>
> >> Since the string being matched is 'sc_in' both the al=
pha and alnum<br>
> >> tokens should have captured it. But we see the following resu=
lt on<br>
> >> execution<br>
> >><br>
> >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D <alnum> Example=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D<br>
> >> Nil<br>
> >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D <alpha> Example=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D<br>
> >> =EF=BD=A2sc_in<foo> bar=EF=BD=A3<br>
> >> ruport =3D> =EF=BD=A2sc_in=EF=BD=A3<br>
> >> type =3D> =EF=BD=A2sc_in=EF=BD=A3<br>
> >> alpha =3D> =EF=BD=A2s=EF=BD=A3<br>
> >> alpha =3D> =EF=BD=A2c=EF=BD=A3<br>
> >> alpha =3D> =EF=BD=A2_=EF=BD=A3<br>
> >> alpha =3D> =EF=BD=A2i=EF=BD=A3<br>
> >> alpha =3D> =EF=BD=A2n=EF=BD=A3<br>
> >><br>
> >><br>
> >> Perl Version is<br>
> >><br>
> >> This is Rakudo Star version 2018.06 built on MoarVM version 2=
018.06<br>
> >> implementing Perl 6.c.<br>
> >><br>
> >><br>
> >><br>
> >> --<br>
> >> Vijayvithal<br>
> >> Dyumnin Semiconductors<br>
> >><br>
> ><br>
> ><br>
> > --<br>
> > brandon s allbery kf8nh<br>
> > <a href=3D"mailto:[email protected]" target=3D"_blank">allbery.=
[email protected]</a><br>
> ><br>
<br>
</blockquote></div>
--0000000000004a40800576f57f12--