Bug #71207 [Opn->Nab]: token_get_all does not recognize context-sensitive keywords

[email protected]
Newsgroups php.bugs
Message-ID <[email protected]>
Edit report at https://bugs.php.net/bug.php?id=71207&edit=1

 ID:                 71207
 Updated by:         [email protected]
 Reported by:        kontakt at kukulich dot cz
-Summary:            token_get_all doesn't work correctly
+Summary:            token_get_all does not recognize context-sensitive
                     keywords
-Status:             Open
+Status:             Not a bug
 Type:               Bug
 Package:            Unknown/Other Function
 PHP Version:        7.0.1
 Block user comment: N
 Private report:     N

 New Comment:

If you want these to be recognized, you must pass the TOKEN_PARSE flag as the second argument to token_get_all().


Previous Comments:
------------------------------------------------------------------------
[2015-12-23 22:33:21] kontakt at kukulich dot cz

Description:
------------
Thanks to https://wiki.php.net/rfc/context_sensitive_lexer it's possible in PHP7 to use a lot of keywords as constants, methods etc.

However token_get_all doesn't return the right tokens.

Test script:
---------------
<?php

$tokens = token_get_all('<?php class Test {const PUBLIC=0;}');

foreach ($tokens as $token) {
	if (is_array($token)) {
		echo token_name($token[0]) . "\n";
	}
}


Expected result:
----------------
T_OPEN_TAG
T_CLASS
T_WHITESPACE
T_STRING
T_WHITESPACE
T_CONST
T_WHITESPACE
T_STRING
T_LNUMBER

Actual result:
--------------
T_OPEN_TAG
T_CLASS
T_WHITESPACE
T_STRING
T_WHITESPACE
T_CONST
T_WHITESPACE
T_PUBLIC
T_LNUMBER


------------------------------------------------------------------------



--
Edit this bug report at https://bugs.php.net/bug.php?id=71207&edit=1
lmpx.com only provides a reader for public news (NNTP) servers. It is not affiliated with the servers or forums shown here and is not responsible for the content of articles, which is written by their respective authors.