Re: Markus King's "UTF-8 decoder capability and stress test"

On 6/9/26 10:48 PM, William Michels via perl6-users wrote:
> https://www.cl.cam.ac.uk/~mgk25/ucs/examples/UTF-8-test.txt <https:// 
> www.cl.cam.ac.uk/~mgk25/ucs/examples/UTF-8-test.txt>
> 
> Unicode/UTF-8 Question:
> 
> Does anyone know if Rakudo has ever been tested against Markus 
> King's "UTF-8 decoder capability and stress test"?
> 
>  From the intro:
> 
>     "This test file can help you examine, how your UTF-8 decoder handles
>     various types of correct, malformed, or otherwise interesting UTF-8
>     sequences."
> 
> 
> Any info appreciated,
> 
> Thx, Bill.


Not sure what you are after, but here is my converter.
C strings are 16 bit and end in two chr(0)'s.  I
convert them with brute force.  I have tried with
the utf16 to utf8 converters and has never happy
with the result.

It presumes a UTF16 little endian C String and converts
to UTF8.


constant BYTE     := uint8;
constant BYTES    := CArray[BYTE];
sub c-to-raku-str( BYTES $CStr ) returns Str  is export( :c-to-raku-str ) {
    # Note: C Strings are always terminated with a nul.  This sub will 
malfunction without it.
    # Note: presumes a UTF16 little endian C String and converts to UTF8

    my Str    $SubName   = &?ROUTINE.name;
    my Str    $RakuStr   = "";
    my Str    $Msg       = "";
    my uint32 $CStrElems = $CStr.elems;
    # say $CStrElems;

    loop (my $i = 0; $i < $CStrElems - 2 ; $i += 2) {
       if  $CStr[ $i ] == 0  &&  $CStr[ $i + 1 ] == 0  { last; }

       if $i == $CStrElems - 4  {
          $Msg = "$SubName ERROR:" ~ " C Strings are required to be 
terminated with a nul\n\n" ~
                 "                     Returning an empty string\n" ~
                 "                     Cowardly existing\n";
          exit;
       }

       # print $CStr[ $i ] ~ "," ~ $CStr[ $i + 1 ] ~ ",  ";
       $RakuStr ~= chr( $CStr[ $i ] );
    }
    # say "";

    # say "$RakuStr";
    return $RakuStr;
}