[PHP-BUG] Bug #71207 [NEW]: token_get_all doesn't work correctly

From:             kontakt at kukulich dot cz
Operating system: 
PHP version:      7.0.1
Package:          Unknown/Other Function
Bug Type:         Bug
Bug description:token_get_all doesn't work correctly

Description:
------------
Thanks to https://wiki.php.net/rfc/context_sensitive_lexer it's possible
in PHP7 to use a lot of keywords as constants, methods etc.

However token_get_all doesn't return the right tokens.

Test script:
---------------
<?php

$tokens = token_get_all('<?php class Test {const PUBLIC=0;}');

foreach ($tokens as $token) {
	if (is_array($token)) {
		echo token_name($token[0]) . "\n";
	}
}


Expected result:
----------------
T_OPEN_TAG
T_CLASS
T_WHITESPACE
T_STRING
T_WHITESPACE
T_CONST
T_WHITESPACE
T_STRING
T_LNUMBER

Actual result:
--------------
T_OPEN_TAG
T_CLASS
T_WHITESPACE
T_STRING
T_WHITESPACE
T_CONST
T_WHITESPACE
T_PUBLIC
T_LNUMBER

-- 
Edit bug report at https://bugs.php.net/bug.php?id=71207&edit=1
-- 
Try a snapshot (PHP 5.4):   https://bugs.php.net/fix.php?id=71207&r=trysnapshot54
Try a snapshot (PHP 5.5):   https://bugs.php.net/fix.php?id=71207&r=trysnapshot55
Try a snapshot (trunk):     https://bugs.php.net/fix.php?id=71207&r=trysnapshottrunk
Fixed in SVN:               https://bugs.php.net/fix.php?id=71207&r=fixed
Fixed in release:           https://bugs.php.net/fix.php?id=71207&r=alreadyfixed
Need backtrace:             https://bugs.php.net/fix.php?id=71207&r=needtrace
Need Reproduce Script:      https://bugs.php.net/fix.php?id=71207&r=needscript
Try newer version:          https://bugs.php.net/fix.php?id=71207&r=oldversion
Not developer issue:        https://bugs.php.net/fix.php?id=71207&r=support
Expected behavior:          https://bugs.php.net/fix.php?id=71207&r=notwrong
Not enough info:            https://bugs.php.net/fix.php?id=71207&r=notenoughinfo
Submitted twice:            https://bugs.php.net/fix.php?id=71207&r=submittedtwice
register_globals:           https://bugs.php.net/fix.php?id=71207&r=globals
PHP 4 support discontinued: https://bugs.php.net/fix.php?id=71207&r=php4
Daylight Savings:           https://bugs.php.net/fix.php?id=71207&r=dst
IIS Stability:              https://bugs.php.net/fix.php?id=71207&r=isapi
Install GNU Sed:            https://bugs.php.net/fix.php?id=71207&r=gnused
Floating point limitations: https://bugs.php.net/fix.php?id=71207&r=float
No Zend Extensions:         https://bugs.php.net/fix.php?id=71207&r=nozend
MySQL Configuration Error:  https://bugs.php.net/fix.php?id=71207&r=mysqlcfg