Converting byte position to character position

"Per Hedbor () @ Pike (-) importm?te f?r mailinglistan" <[email protected]>
Newsgroups gmane.comp.lang.pike.user
Message-ID <[email protected]>
> And it works, but ow! what a mess. I could alternatively go through
> the string manually, counting up by byte range (<127 = add one, <0x800
> = add two, etc), but I suspect that'll work out even messier. Is there
> a cleaner way to do this?

Well. With UTF-8, no, not really. Unless you cache it somewhere.

This version is almost exactly as fast for short strings (like your
example) but significantly faster for long strings:

int utf8byte_to_char( string txt, int pos )
{
   int m = String.range(txt)[1];
   if( m < 128 )
     return pos;
   int c;
   if( m < 0x800 )
   {
      for( ; pos; c++,pos-- )
       if( txt[c] > 0x80 ) pos--;
   }
   else if( m < 0x10000 )
   {
      for( ; pos; c++,pos-- ) {
       if( txt[c] > 0x800 ) pos--;
       if( txt[c] > 0x80 ) pos--;
     }
   }
   else for( ; pos; c++,pos-- )
   {
     switch( txt[c] )
     {
	case 0x4000000..: pos--;
	case 0x200000..0x3ffffff: pos--;
	case 0x10000..0x1ffff: pos--;
	case 0x800..0xffff: pos--;
	case 0x80..0x7ff: pos--;
     }
   }
   return c;
}

Of course, your version can also be optimized significantly, at least
when the desired position is rather small and the string is long:

int utf8byte_to_char2( string txt int pos )
{
   return  sizeof(utf8_to_string(string_to_utf8(txt[..pos-1])[..pos-1]));
}

This is the fastest one I have tested yet, but please avoid looking
for byte 10000394 in a 2Mchar long unicode string... :)

-- 
Per Hedbor
lmpx.com only provides a reader for public news (NNTP) servers. It is not affiliated with the servers or forums shown here and is not responsible for the content of articles, which is written by their respective authors.