Re: cvs: ZendEngine2(PHP_5_3) / Zend.m4 zend.c zend_compile.c zend_globals.h zend_highlight.c zend_language_scanner.c zend_language_scanner.h zend_language_scanner.l zend_language_scanner_defs.h zend_multibyte.h

Hello Rui,

Sunday, July 6, 2008, 3:31:40 AM, you wrote:

> Hello Marcus,

> On Sat, 5 Jul 2008 20:46:33 +0200
> Marcus Boerger <[email protected]> wrote:

>> Hello Rui,
>> 
>>   briefly scanning to this you took a complete different approach I did.
>> While used newer re2c features to generate two different scanners you seem
>> to have reapplied the the old code.  That would mean you're most likely a
>> bit slower but have less code too. 
>> Did you change anything or did you
>> simply reapply one by one? If you did just copy I might spend some time
>> appliying a few fixes I encountered. 

> Thank you for comments. 
> My implementation is ported code from PHP 5.2 with minimal change
> because I am not familiar with lexical analizer including re2c, flex.
> I read man page of re2c on the web site, and Peter's paper,
> but, I couldn't understand the detail.


> What we need to define for multibyte encoding support is 
> how to treat some flex incompatible encodings like Shift_JIS.

> The implementation is like,
> 1. read script from file system.
> 2. convert from script encoding to flex compatible encoding
> 3. applied to lexer
> 4.convert from flex compatible encoding to internal encoding (if
> necessary)

> The script encoding is specified by mbstinrg.scirpt_encoding 
> in php.ini , the script file itself as BOM in Unicode (like UTF-8, UTF16), 
> in the script file using 'declare'.

> The approach used in PHP 5.2 might be ineffective for re2c.
> I will appreciate if you make a better implementaion based on
> the advanced feature of re2c.

>> Also we are still missing tests.
>> Without tests I feel we have to remove all of this anyway.

> Currently,  a couple of test scripts is already existing in
> Zend/tests/multibyte/*.phpt

> 'still missing' means we should make a huge number of test scripts
> or a higher percentage of code coverage is necessary ?

> The encoding converstion engine part is already tested
> as mbstring extension.
> The scanner part including encoding conversion, encoding
> detection should be tested.

> The current test scipts are covering the most important 
> feature of zend-multibyte support,

> 1. test for flex incompatible encoding (0x5c issue) and 'declare' sentence
>   multibye_encoding_001.phpt

We don't have a test yet that switches encodings with multiple declare
statements or should that be disallowed?

> 2. test for UTF-8 BOM
>   multibye_encoding_002.phpt
> 3. test for UTF-16 BOM and UTF-16 encoded Unicode script
>   multibye_encoding_003.phpt

> The encoding conversion feature is also covered by these scripts.
> What kind of test scripts do you need ?

Anything that brings code coverage for the multibyte stuff as high as
possible.


Best regards,
 Marcus


-- 
Zend Engine CVS Mailing List (http://cvs.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php