Re: [perl #133541] Grammer bug <alnum> vs <alpha>

--6znutc32dikpqm2w
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

This issue surfaces because of the token TOP line. If instead of
<rport>|<ruport> only ruport was used the testcase works for both cases. So=
 it is quite
possible that the bug is elsewhere but shows up as a difference between
alpha and alnum.

Regards
Vijay


On Fri, Sep 28, 2018 at 07:18:49AM -0700, Patrick R. Michaud via RT wrote:
> The issue doesn't seem to be the underscore, because I get the same resul=
t even when converting the underscore into a letter ('b'):
>=20
> $ cat gentb.p6=20
> grammar G0 {
> 	token TOP {<rport>|<ruport>.*}
> 	regex rport { <type>}
> 	rule ruport { <type>}
> 	#token type {<alpha>+}
> 	token type {<alnum>+}
> }
>=20
> grammar G1 {
> 	token TOP {<rport>|<ruport>.*}
> 	regex rport { <type>}
> 	rule ruport { <type>}
> 	token type {<alpha>+}
> 	#token type {<alnum>+}
> }
> my $str=3D"scbin<foo> bar";
> say "=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D <alnum> Example=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D";
> say G0.parse($str);
> say "=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D <alpha> Example=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D";
> say G1.parse($str);
>=20
> $ perl6 gentb.p6
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D <alnum> Example=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D
> Nil
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D <alpha> Example=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D
> =EF=BD=A2scbin<foo> bar=EF=BD=A3
>  ruport =3D> =EF=BD=A2scbin=EF=BD=A3
>   type =3D> =EF=BD=A2scbin=EF=BD=A3
>    alpha =3D> =EF=BD=A2s=EF=BD=A3
>    alpha =3D> =EF=BD=A2c=EF=BD=A3
>    alpha =3D> =EF=BD=A2b=EF=BD=A3
>    alpha =3D> =EF=BD=A2i=EF=BD=A3
>    alpha =3D> =EF=BD=A2n=EF=BD=A3
> $=20
>=20
>=20
> On Fri, Sep 28, 2018 at 02:26:41AM -0700, Brent Laabs wrote:
> > Are you sure about that?  Underscore has been part of the specs (synops=
es)
> > for <alpha> for at least 10 years, probably longer.
> >=20
> >  >  "_" ~~ /<alpha>/
> > =EF=BD=A2_=EF=BD=A3
> >  alpha =3D> =EF=BD=A2_=EF=BD=A3
> >=20
> > On Thu, Sep 27, 2018 at 7:52 PM Brandon Allbery <[email protected]> w=
rote:
> >=20
> > > "_" is not an alphabetic character. It's allowed in "alnum" because t=
hat
> > > is by intent what is \w in other regex implementations, which include=
s "_".
> > >
> > > On Thu, Sep 27, 2018 at 10:47 PM Vijayvithal <perl6-bugs-followup@per=
l.org>
> > > wrote:
> > >
> > >> # New Ticket Created by  Vijayvithal
> > >> # Please include the string:  [perl #133541]
> > >> # in the subject line of all future correspondence about this issue.
> > >> # <URL: https://rt.perl.org/Ticket/Display.html?id=3D133541 >
> > >>
> > >>
> > >> In the attached code, the only difference between the Grammars G0 an=
d G1
> > >> is the defination of token 'type' it is defined as <alpha> in one ca=
se
> > >> and as <alnum> in another.
> > >>
> > >> Since the string being matched is 'sc_in' both the alpha and alnum
> > >> tokens should have captured it. But we see the following result on
> > >> execution
> > >>
> > >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D <alnum> Example=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D
> > >> Nil
> > >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D <alpha> Example=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D
> > >> =EF=BD=A2sc_in<foo> bar=EF=BD=A3
> > >> ruport =3D> =EF=BD=A2sc_in=EF=BD=A3
> > >> type =3D> =EF=BD=A2sc_in=EF=BD=A3
> > >> alpha =3D> =EF=BD=A2s=EF=BD=A3
> > >> alpha =3D> =EF=BD=A2c=EF=BD=A3
> > >> alpha =3D> =EF=BD=A2_=EF=BD=A3
> > >> alpha =3D> =EF=BD=A2i=EF=BD=A3
> > >> alpha =3D> =EF=BD=A2n=EF=BD=A3
> > >>
> > >>
> > >> Perl Version is
> > >>
> > >> This is Rakudo Star version 2018.06 built on MoarVM version 2018.06
> > >> implementing Perl 6.c.
> > >>
> > >>
> > >>
> > >> --
> > >> Vijayvithal
> > >> Dyumnin Semiconductors
> > >>
> > >
> > >
> > > --
> > > brandon s allbery kf8nh
> > > [email protected]
> > >
>=20

--=20
Vijayvithal
Dyumnin Semiconductors

--6znutc32dikpqm2w
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)

iQEcBAEBAgAGBQJbrkPXAAoJEKAKZ9Q2gpUZm80H/i6fPk2ODrI0OL7X/9UTTs70
iME7uSysC5RrwJcvdlY5j6NSP72SY988LmWhgq8UgEF2D6ilpHwV0NxrSjUQdmPJ
3IMx219fekfcsIdzrymd7sVE/LH4E6J4JbHrlzryqFpNxtoPJkXrT2A35ng2/0N4
aCKotlTXS07FPx6S7HxNA6Rw6zNSuznp31aYeddQWqlYGtBmcsPESYtkIIq19Ldu
2+aeJ6ZUoirTyTrkl/P634Q6eZV5V3fXN812rjjseKyGgFkGwh12Y4sU4phXlxQY
t96V8mdAudehKqT2f/uw7mDuofDJpPirY69WS4VdDJ2X0G/OXSdrLnX+Ac493fg=
=lohU
-----END PGP SIGNATURE-----

--6znutc32dikpqm2w--