[perl #133541] Grammer bug <alnum> vs <alpha>

------------=_1537874196-22296-0
Content-Transfer-Encoding: 8bit
Content-Type: text/plain; charset="utf-8"

# New Ticket Created by  Vijayvithal 
# Please include the string:  [perl #133541]
# in the subject line of all future correspondence about this issue. 
# <URL: https://rt.perl.org/Ticket/Display.html?id=133541 >


In the attached code, the only difference between the Grammars G0 and G1
is the defination of token 'type' it is defined as <alpha> in one case
and as <alnum> in another.

Since the string being matched is 'sc_in' both the alpha and alnum
tokens should have captured it. But we see the following result on
execution

=========== <alnum> Example==============
Nil
=========== <alpha> Example==============
｢sc_in<foo> bar｣
ruport => ｢sc_in｣
type => ｢sc_in｣
alpha => ｢s｣
alpha => ｢c｣
alpha => ｢_｣
alpha => ｢i｣
alpha => ｢n｣


Perl Version is 

This is Rakudo Star version 2018.06 built on MoarVM version 2018.06
implementing Perl 6.c.



-- 
Vijayvithal
Dyumnin Semiconductors

------------=_1537874196-22296-0
Content-Type: text/plain; charset="ascii"; name="gentb.p6"
Content-Disposition: attachment; filename="gentb.p6"
Content-Transfer-Encoding: 7bit
RT-Attachment: 133541/1583606/827771

grammar G0 {
	token TOP {<rport>|<ruport>.*}
	regex rport { <type>}
	rule ruport { <type>}
	#token type {<alpha>+}
	token type {<alnum>+}
}

grammar G1 {
	token TOP {<rport>|<ruport>.*}
	regex rport { <type>}
	rule ruport { <type>}
	token type {<alpha>+}
	#token type {<alnum>+}
}
my $str="sc_in<foo> bar";
say "=========== <alnum> Example==============";
say G0.parse($str);
say "=========== <alpha> Example==============";
say G1.parse($str);

------------=_1537874196-22296-0
Content-Type: application/pgp-signature; name="signature.asc"
Content-Disposition: attachment; filename="signature.asc"
Content-Transfer-Encoding: base64
RT-Attachment: 133541/1583606/827772

LS0tLS1CRUdJTiBQR1AgU0lHTkFUVVJFLS0tLS0KVmVyc2lvbjogR251UEcg
djIuMC4yMiAoR05VL0xpbnV4KQoKaVFFY0JBRUJBZ0FHQlFKYnFoY1NBQW9K
RUtBS1o5UTJncFVaN2VzSC8yS2svbDg0ZUEvaEZpajRyNVFtem9Fbgpnb0sr
MktDY1pyckpnQkxyT0xJUFFPOE1QR0l5V21UaVoxcktnVVFYakladFBBOUJT
by9ueUM4Zm5pU2RRRENvCkxJOU54VXRtVjZiUkY5V09CMjJId2puTmMwTUov
QVRxd3VmMW9NTnNRWnYyWjlTbEphKzVEa01qSGF1MlBBU2YKYkhhS0hZTmVj
WGF5eUY5endoMGFMMjRNcW5EMksxT0RLbWZGWjhqNk5KWWsrd0IyVzUyaGxa
S1hpT29oU1FIOAora3ZRRzVFT05GSUQwM041ZTlBa2NXYTBBR3MyRXh0N29a
RUhvVlFIQXNjYkcwNFdNVDVZM2xocG1taWJKZFhSCk4wSjJZaFJIMG1rZXJr
RFdzQ2FYV1lLRTBQUFVrbkFDR1JZTDJHZ2xSL1VDbldQNGNvUWV6QmhxeU9Y
N1JDbz0KPXV1VFAKLS0tLS1FTkQgUEdQIFNJR05BVFVSRS0tLS0tCg==

------------=_1537874196-22296-0--