Re: [idn] Re: FYI: BOF on Internationalized Email Addresses (IEA)

I stand corrected (and sit surprised).

Mark
__________________________________
http://www.macchiato.com
► शिष्यादिच्छेत्पराजयम् ◄

----- Original Message ----- 
From: "Adam M. Costello" <[email protected]>
To: "IETF IMAA list" <[email protected]>
Sent: Wed, 2003 Oct 29 18:20
Subject: Re: [idn] Re: FYI: BOF on Internationalized Email Addresses (IEA)


>
> Mark Davis <[email protected]> wrote:
>
> > And this is, of course, only for short strings.  A 10K file of
> > Cyrillic converted to Punycode would blow out completely.
>
> Think so?  Let's find out...
>
> The easiest way to find a bunch of Russian text was to search Google
> for "bible russian", so I grabbed the first five chapters of Genesis
> (represented in koi8-r, an 8-bit charset) and piped them through "tr -d
> '[\000-\177]'" to remove all ASCII characters, leaving 10226 Cyrillic
> characters.
>
> I then used "recode koi8-r..ucs-4" and a homemade script to convert
> those characters to U+ notation.
>
> I then edited the Punycode sample implementation to increase the static
> array sizes of the input and output to 11000 and 22000 respectively.
>
> The UTF-8 encoding of the text is 20452 bytes.  The Punycode encoding is
> 11970 caseless ASCII letters and digits (base 36).
>
> But for a string that large, conventional compression techniques work
> better than Punycode's technique.  Compressing the UTF-8 encoding with
> gzip, then converting to base-32, results in 8332 ASCII letters and
> digits.
>
> Summary:
>
>           input:  10226 Cyrillic characters
>           UTF-8:  20452 bytes (high bit always set)
>        Punycode:  11970 base-36 ASCII characters
> UTF-8.gz.base32:   8332 base-32 ASCII characters
>
> I think with strings beyond about 4000 characters, Punycode starts
> to run the risk of bumping up against the limit of 32-bit integers.
> Although strings that use only plane 0 should be fine up to around
> 60,000 characters.
>
> The O(n^2) algorithm given in the Punycode spec is costly for long
> strings (my 864 MHz Pentium III could encode the 10k test string only 44
> times per second).  I'm pretty sure I can write an O(n log n) algorithm
> for Punycode, but I haven't actually done it yet.
>
> Of course you wouldn't want to use Punycode for long strings anyway,
> you'd want to use conventional compression that exploits repeated
> substrings, like deflate.  But maybe the O(n log n) algorithm will
> be faster than the O(n^2) algorithm for medium-length strings, where
> Punycode still compresses better than deflate.
>
> AMC
>