Re: [perl #133541] Grammer bug <alnum> vs <alpha>
[email protected] (Brent Laabs) Mon, 1 Oct 2018 19:51:37 -0700
| Newsgroups | perl.perl6.compiler |
|---|---|
| Message-ID | <CAAPDpBxAR9K2CwimGzS=SWnTB85co3O1++9JK0Cr-k+H2ux78g@mail.gmail.com> |
--000000000000cd38e90577360044 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Actually, if you change it to <ruport>.*|<rport> -- this will work as you expect. It's a bug that your version doesn't work, of course. It does seem to involve <alpha> tangentially, but it is unrelated to underscore. On Mon, Oct 1, 2018 at 6:17 PM Vijayvithal via RT < [email protected]> wrote: > This issue surfaces because of the token TOP line. If instead of > <rport>|<ruport> only ruport was used the testcase works for both cases. > So it is quite > possible that the bug is elsewhere but shows up as a difference between > alpha and alnum. > > Regards > Vijay > > > On Fri, Sep 28, 2018 at 07:18:49AM -0700, Patrick R. Michaud via RT wrote= : > > The issue doesn't seem to be the underscore, because I get the same > result even when converting the underscore into a letter ('b'): > > > > $ cat gentb.p6 > > grammar G0 { > > token TOP {<rport>|<ruport>.*} > > regex rport { <type>} > > rule ruport { <type>} > > #token type {<alpha>+} > > token type {<alnum>+} > > } > > > > grammar G1 { > > token TOP {<rport>|<ruport>.*} > > regex rport { <type>} > > rule ruport { <type>} > > token type {<alpha>+} > > #token type {<alnum>+} > > } > > my $str=3D"scbin<foo> bar"; > > say "=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D <alnum> Example=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D"; > > say G0.parse($str); > > say "=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D <alpha> Example=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D"; > > say G1.parse($str); > > > > $ perl6 gentb.p6 > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D <alnum> Example=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D > > Nil > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D <alpha> Example=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D > > =EF=BD=A2scbin<foo> bar=EF=BD=A3 > > ruport =3D> =EF=BD=A2scbin=EF=BD=A3 > > type =3D> =EF=BD=A2scbin=EF=BD=A3 > > alpha =3D> =EF=BD=A2s=EF=BD=A3 > > alpha =3D> =EF=BD=A2c=EF=BD=A3 > > alpha =3D> =EF=BD=A2b=EF=BD=A3 > > alpha =3D> =EF=BD=A2i=EF=BD=A3 > > alpha =3D> =EF=BD=A2n=EF=BD=A3 > > $ > > > > > > On Fri, Sep 28, 2018 at 02:26:41AM -0700, Brent Laabs wrote: > > > Are you sure about that? Underscore has been part of the specs > (synopses) > > > for <alpha> for at least 10 years, probably longer. > > > > > > > "_" ~~ /<alpha>/ > > > =EF=BD=A2_=EF=BD=A3 > > > alpha =3D> =EF=BD=A2_=EF=BD=A3 > > > > > > On Thu, Sep 27, 2018 at 7:52 PM Brandon Allbery <[email protected]> > wrote: > > > > > > > "_" is not an alphabetic character. It's allowed in "alnum" because > that > > > > is by intent what is \w in other regex implementations, which > includes "_". > > > > > > > > On Thu, Sep 27, 2018 at 10:47 PM Vijayvithal < > [email protected]> > > > > wrote: > > > > > > > >> # New Ticket Created by Vijayvithal > > > >> # Please include the string: [perl #133541] > > > >> # in the subject line of all future correspondence about this issu= e. > > > >> # <URL: https://rt.perl.org/Ticket/Display.html?id=3D133541 > > > > >> > > > >> > > > >> In the attached code, the only difference between the Grammars G0 > and G1 > > > >> is the defination of token 'type' it is defined as <alpha> in one > case > > > >> and as <alnum> in another. > > > >> > > > >> Since the string being matched is 'sc_in' both the alpha and alnum > > > >> tokens should have captured it. But we see the following result on > > > >> execution > > > >> > > > >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D <alnum> Example=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D > > > >> Nil > > > >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D <alpha> Example=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D > > > >> =EF=BD=A2sc_in<foo> bar=EF=BD=A3 > > > >> ruport =3D> =EF=BD=A2sc_in=EF=BD=A3 > > > >> type =3D> =EF=BD=A2sc_in=EF=BD=A3 > > > >> alpha =3D> =EF=BD=A2s=EF=BD=A3 > > > >> alpha =3D> =EF=BD=A2c=EF=BD=A3 > > > >> alpha =3D> =EF=BD=A2_=EF=BD=A3 > > > >> alpha =3D> =EF=BD=A2i=EF=BD=A3 > > > >> alpha =3D> =EF=BD=A2n=EF=BD=A3 > > > >> > > > >> > > > >> Perl Version is > > > >> > > > >> This is Rakudo Star version 2018.06 built on MoarVM version 2018.0= 6 > > > >> implementing Perl 6.c. > > > >> > > > >> > > > >> > > > >> -- > > > >> Vijayvithal > > > >> Dyumnin Semiconductors > > > >> > > > > > > > > > > > > -- > > > > brandon s allbery kf8nh > > > > [email protected] > > > > > > > > -- > Vijayvithal > Dyumnin Semiconductors > > --000000000000cd38e90577360044 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr">Actually, if you change it to <ruport>.*|<rport&g= t; -- this will work as you expect.=C2=A0 It's a bug that your version = doesn't work, of course.=C2=A0 It does seem to involve <alpha> ta= ngentially, but it is unrelated to underscore.<br><div><br><div class=3D"gm= ail_quote"><div dir=3D"ltr">On Mon, Oct 1, 2018 at 6:17 PM Vijayvithal via = RT <<a href=3D"mailto:[email protected]">perl6-bugs-followup@= perl.org</a>> wrote:<br></div><blockquote class=3D"gmail_quote" style=3D= "margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">This issue = surfaces because of the token TOP line. If instead of<br> <rport>|<ruport> only ruport was used the testcase works for bo= th cases. So it is quite<br> possible that the bug is elsewhere but shows up as a difference between<br> alpha and alnum.<br> <br> Regards<br> Vijay<br> <br> <br> On Fri, Sep 28, 2018 at 07:18:49AM -0700, Patrick R. Michaud via RT wrote:<= br> > The issue doesn't seem to be the underscore, because I get the sam= e result even when converting the underscore into a letter ('b'):<b= r> > <br> > $ cat gentb.p6 <br> > grammar G0 {<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0token TOP {<rport>|<ruport>.*}<b= r> >=C2=A0 =C2=A0 =C2=A0 =C2=A0regex rport { <type>}<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0rule ruport { <type>}<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0#token type {<alpha>+}<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0token type {<alnum>+}<br> > }<br> > <br> > grammar G1 {<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0token TOP {<rport>|<ruport>.*}<b= r> >=C2=A0 =C2=A0 =C2=A0 =C2=A0regex rport { <type>}<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0rule ruport { <type>}<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0token type {<alpha>+}<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0#token type {<alnum>+}<br> > }<br> > my $str=3D"scbin<foo> bar";<br> > say "=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D <alnum> Example=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D";<br> > say G0.parse($str);<br> > say "=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D <alpha> Example=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D";<br> > say G1.parse($str);<br> > <br> > $ perl6 gentb.p6<br> > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D <alnum> Example=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D<br> > Nil<br> > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D <alpha> Example=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D<br> > =EF=BD=A2scbin<foo> bar=EF=BD=A3<br> >=C2=A0 ruport =3D> =EF=BD=A2scbin=EF=BD=A3<br> >=C2=A0 =C2=A0type =3D> =EF=BD=A2scbin=EF=BD=A3<br> >=C2=A0 =C2=A0 alpha =3D> =EF=BD=A2s=EF=BD=A3<br> >=C2=A0 =C2=A0 alpha =3D> =EF=BD=A2c=EF=BD=A3<br> >=C2=A0 =C2=A0 alpha =3D> =EF=BD=A2b=EF=BD=A3<br> >=C2=A0 =C2=A0 alpha =3D> =EF=BD=A2i=EF=BD=A3<br> >=C2=A0 =C2=A0 alpha =3D> =EF=BD=A2n=EF=BD=A3<br> > $ <br> > <br> > <br> > On Fri, Sep 28, 2018 at 02:26:41AM -0700, Brent Laabs wrote:<br> > > Are you sure about that?=C2=A0 Underscore has been part of the sp= ecs (synopses)<br> > > for <alpha> for at least 10 years, probably longer.<br> > > <br> > >=C2=A0 >=C2=A0 "_" ~~ /<alpha>/<br> > > =EF=BD=A2_=EF=BD=A3<br> > >=C2=A0 alpha =3D> =EF=BD=A2_=EF=BD=A3<br> > > <br> > > On Thu, Sep 27, 2018 at 7:52 PM Brandon Allbery <<a href=3D"ma= ilto:[email protected]" target=3D"_blank">[email protected]</a>> wro= te:<br> > > <br> > > > "_" is not an alphabetic character. It's allow= ed in "alnum" because that<br> > > > is by intent what is \w in other regex implementations, whic= h includes "_".<br> > > ><br> > > > On Thu, Sep 27, 2018 at 10:47 PM Vijayvithal <<a href=3D"= mailto:[email protected]" target=3D"_blank">perl6-bugs-followup@= perl.org</a>><br> > > > wrote:<br> > > ><br> > > >> # New Ticket Created by=C2=A0 Vijayvithal<br> > > >> # Please include the string:=C2=A0 [perl #133541]<br> > > >> # in the subject line of all future correspondence about= this issue.<br> > > >> # <URL: <a href=3D"https://rt.perl.org/Ticket/Display= .html?id=3D133541" rel=3D"noreferrer" target=3D"_blank">https://rt.perl.org= /Ticket/Display.html?id=3D133541</a> ><br> > > >><br> > > >><br> > > >> In the attached code, the only difference between the Gr= ammars G0 and G1<br> > > >> is the defination of token 'type' it is defined = as <alpha> in one case<br> > > >> and as <alnum> in another.<br> > > >><br> > > >> Since the string being matched is 'sc_in' both t= he alpha and alnum<br> > > >> tokens should have captured it. But we see the following= result on<br> > > >> execution<br> > > >><br> > > >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D <alnum> Example= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D<br> > > >> Nil<br> > > >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D <alpha> Example= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D<br> > > >> =EF=BD=A2sc_in<foo> bar=EF=BD=A3<br> > > >> ruport =3D> =EF=BD=A2sc_in=EF=BD=A3<br> > > >> type =3D> =EF=BD=A2sc_in=EF=BD=A3<br> > > >> alpha =3D> =EF=BD=A2s=EF=BD=A3<br> > > >> alpha =3D> =EF=BD=A2c=EF=BD=A3<br> > > >> alpha =3D> =EF=BD=A2_=EF=BD=A3<br> > > >> alpha =3D> =EF=BD=A2i=EF=BD=A3<br> > > >> alpha =3D> =EF=BD=A2n=EF=BD=A3<br> > > >><br> > > >><br> > > >> Perl Version is<br> > > >><br> > > >> This is Rakudo Star version 2018.06 built on MoarVM vers= ion 2018.06<br> > > >> implementing Perl 6.c.<br> > > >><br> > > >><br> > > >><br> > > >> --<br> > > >> Vijayvithal<br> > > >> Dyumnin Semiconductors<br> > > >><br> > > ><br> > > ><br> > > > --<br> > > > brandon s allbery kf8nh<br> > > > <a href=3D"mailto:[email protected]" target=3D"_blank">all= [email protected]</a><br> > > ><br> > <br> <br> -- <br> Vijayvithal<br> Dyumnin Semiconductors<br> <br> </blockquote></div></div></div> --000000000000cd38e90577360044--