[PHP-BUG] Bug #78978 [NEW]: Tokenizer tokenizes hex numbers as floats on Windows

From:             bugs dot php dot net_nospam at adviesenzo dot nl
Operating system: Windows 7
PHP version:      7.4.0
Package:          PHP Language Specification
Bug Type:         Bug
Bug description:Tokenizer tokenizes hex numbers as floats on Windows

Description:
------------
Noticed this while working on something for PHP_CodeSniffer where I
wasn't getting the same results for unit tests on Travis versus my local
Windows machine.

It appears that hexidecimal integers using the new PHP 7.4 numeric
literal separator are tokenized differently on Linux versus Windows.

They should be tokenized as `T_LNUMBER` and are on Linux. See:
https://3v4l.org/vlH5j

However on Windows, I'm seeing them get tokenized as `T_DNUMBER`.







Test script:
---------------
$tokens = token_get_all(
	'<?php
$foo = 0xCAFE_F00D;
$foo = 0x42_72_6F_77_6E;'
);

foreach ($tokens as $token) {
    if (is_array($token)) {
        echo "Line {$token[2]}: ", token_name($token[0]), "
('{$token[1]}')", PHP_EOL;
    }
}

Expected result:
----------------
Line 1: T_OPEN_TAG ('<?php
')
Line 2: T_VARIABLE ('$foo')
Line 2: T_WHITESPACE (' ')
Line 2: T_WHITESPACE (' ')
Line 2: T_LNUMBER ('0xCAFE_F00D')
Line 2: T_WHITESPACE (' ')
Line 3: T_VARIABLE ('$foo')
Line 3: T_WHITESPACE (' ')
Line 3: T_WHITESPACE (' ')
Line 3: T_LNUMBER ('0x42_72_6F_77_6E')

Actual result:
--------------
Line 1: T_OPEN_TAG ('<?php
')
Line 2: T_VARIABLE ('$foo')
Line 2: T_WHITESPACE (' ')
Line 2: T_WHITESPACE (' ')
Line 2: T_DNUMBER ('0xCAFE_F00D')
Line 2: T_WHITESPACE ('
')
Line 3: T_VARIABLE ('$foo')
Line 3: T_WHITESPACE (' ')
Line 3: T_WHITESPACE (' ')
Line 3: T_DNUMBER ('0x42_72_6F_77_6E')

-- 
Edit bug report at https://bugs.php.net/bug.php?id=78978&edit=1
-- 
Fix committed:                    https://bugs.php.net/fix.php?id=78978&r=fixed
Fixed in release:                 https://bugs.php.net/fix.php?id=78978&r=alreadyfixed
Need backtrace:                   https://bugs.php.net/fix.php?id=78978&r=needtrace
Need Reproduce Script:            https://bugs.php.net/fix.php?id=78978&r=needscript
Try newer version:                https://bugs.php.net/fix.php?id=78978&r=oldversion
Not developer issue:              https://bugs.php.net/fix.php?id=78978&r=support
Expected behavior:                https://bugs.php.net/fix.php?id=78978&r=notwrong
Not enough info:                  https://bugs.php.net/fix.php?id=78978&r=notenoughinfo
Submitted twice:                  https://bugs.php.net/fix.php?id=78978&r=submittedtwice
register_globals:                 https://bugs.php.net/fix.php?id=78978&r=globals
PHP version support discontinued: https://bugs.php.net/fix.php?id=78978&r=phptooold
Daylight Savings:                 https://bugs.php.net/fix.php?id=78978&r=dst
IIS Stability:                    https://bugs.php.net/fix.php?id=78978&r=isapi
Install GNU Sed:                  https://bugs.php.net/fix.php?id=78978&r=gnused
Floating point limitations:       https://bugs.php.net/fix.php?id=78978&r=float
No Zend Extensions:               https://bugs.php.net/fix.php?id=78978&r=nozend
MySQL Configuration Error:        https://bugs.php.net/fix.php?id=78978&r=mysqlcfg