Re: [idn] Re: FYI: BOF on Internationalized Email Addresses (IEA)

Mark Davis <[email protected]> wrote:

> And this is, of course, only for short strings.  A 10K file of
> Cyrillic converted to Punycode would blow out completely.

Think so?  Let's find out...

The easiest way to find a bunch of Russian text was to search Google
for "bible russian", so I grabbed the first five chapters of Genesis
(represented in koi8-r, an 8-bit charset) and piped them through "tr -d
'[\000-\177]'" to remove all ASCII characters, leaving 10226 Cyrillic
characters.

I then used "recode koi8-r..ucs-4" and a homemade script to convert
those characters to U+ notation.

I then edited the Punycode sample implementation to increase the static
array sizes of the input and output to 11000 and 22000 respectively.

The UTF-8 encoding of the text is 20452 bytes.  The Punycode encoding is
11970 caseless ASCII letters and digits (base 36).

But for a string that large, conventional compression techniques work
better than Punycode's technique.  Compressing the UTF-8 encoding with
gzip, then converting to base-32, results in 8332 ASCII letters and
digits.

Summary:

          input:  10226 Cyrillic characters
          UTF-8:  20452 bytes (high bit always set)
       Punycode:  11970 base-36 ASCII characters
UTF-8.gz.base32:   8332 base-32 ASCII characters

I think with strings beyond about 4000 characters, Punycode starts
to run the risk of bumping up against the limit of 32-bit integers.
Although strings that use only plane 0 should be fine up to around
60,000 characters.

The O(n^2) algorithm given in the Punycode spec is costly for long
strings (my 864 MHz Pentium III could encode the 10k test string only 44
times per second).  I'm pretty sure I can write an O(n log n) algorithm
for Punycode, but I haven't actually done it yet.

Of course you wouldn't want to use Punycode for long strings anyway,
you'd want to use conventional compression that exploits repeated
substrings, like deflate.  But maybe the O(n log n) algorithm will
be faster than the O(n^2) algorithm for medium-length strings, where
Punycode still compresses better than deflate.

AMC