Re: Domain check in draft-obispo-epp-idn-00.txt

On 04/01/12 20:37, Michael Young wrote:

> What can I say, it would be great if the registrar put that kind of intelligence in their storefront and didn't send domain creates that violate a given language table.

I agree completely!

> Most registrars aren't making that extensive of an effort. Why?

I think I can shed some light on that, and there are a number of ways that
registries can help registrars with this.

I spent a good amount of time hacking together a set of libraries and tools
to help deal with it on our end. The key was to find a way of representing
the whitelists that (a) wouldn't consume a tonne of memory, (b) would be quick
to unmarshall from disc, (c) would work effectively with both alphabetic
scripts and non-alphabetic scripts, and (d) would have minimal CPU overhead.
After a number of experiments I hit upon compiling the codepoint whitelist
into the following format[1]:

First, I imagined each Unicode plane as divided up into 256 slices. This was
a useful as it corresponds pretty closely with how codepoints were allocated
in the first plane at least, so each slice is typically going to either empty
or fairly densely packed with whitelisted codepoints. I used 256b bitvectors
to represent non-empty slices, along with a single bitvector with all the bits
off for slices with no whitelisted characters.

Next, for each slice in a plane, I stored the index of a bitvector that
corresponds to the whitelisted codepoints within that slice. In addition, it
maintains a bitfield of the planes represented within the table. This allowed
for a compact representation of the whitelisted CJK codepoints in the higher
planes (especially for Chinese) while having only a minor impact on the size
of tables for languages represented BMP and SMP. It also kept the speed of
unmarshalling low.

Finally, there's an optimisation stop to remove any duplicate non-empty
slices. This was particularly useful for Korean.

Each table is tagged with a language code and compiled into a set of files,
one for each registry.

Now, I chose this way of doing things as I needed to be able to load those
tables repeatedly into memory from disc from various languages, including C,
Python and PHP. If our systems were purely written in C and Python, I'd have
just loaded textual representations of the whitelists into memory, but with
PHP, compiling the tables into form that can be quickly unmarshalled was a
huge win.

Getting this right took a lot of thought, however, and the registrar business
is very low margin, so the kind of near obsessive attention to detail that I
showed with this isn't going to be common amongst registrar because acting
fast and loose is more cost-effective.

If you want registrars to behave like you'd prefer (which is the way they
ought to be acting anyway), you have to make it cost them: a demerit system
would help. If a registrar repeatedly sends IDNs to either the <domain:check>
or <domain:create> commands that don't match any of the registry's whitelists,
that registrar ought to be penalised.

As a carrot, registries have to make it easy for registrar to build and
maintain whitelists. IANA's Repository of IDN Practices solves half the
problem here, but it's still deficient. The tables provided typically aren't
trivially parseable. Some, such as Telnic's, are fine, but some of the HTML
codepoint table representations seem as if they were designed to frustrate
ease of parsing. And don't get me started on the tables submitted as PDFs!
It'd also help if IANA provided an Atom feed listing additions and revisions
to the repository so that registrars could automatically parse and download.

More registry operators should publish their whitelists in IANA's practices
repository, and Afilias for one have yet to register anything other than their
German whitelist with IANA, even after over five years of supporting seven
other languages besides German. These things are hardly business secrets,
after all, so I can't see why nobody's got around to emailing the things into
IANA yet, nor can I understand why anybody should have to log into the
registrar relations area to get their hands on them.

> Adjust the domain check ext so the language tag element is optional. No
> language tag and it assumes a straight ASCII registration.

> Then those registrars that want to validate their own work can and
> registrars that aren't sure can test a punycode string against the extended
> domain check. This seems like a compromise that works, thoughts?

That's not really a compromise: that's the status quo of requiring a language
code when checking availability.

Here's a compromise that works for me: registrars should be allowed to submit
straight ASCII domains and IDN domains to both the <domain:check> and
<domain:create> commands, but registries should ensure that the domain is
valid against their codepoint whitelists. If a registrar submits domains to
either command that don't match at least one whitelist, they should be marked
with a demerit and the request rejected. Once a registrar reaches a certain
threshold, the registry should notify the offending registrar that they risk
sanction for submitting bad IDNs. Once they hit a higher threshold, they
should should be notified that they have been penalised for their behaviour.
The form of penalisation should be at the registries discretion, and might
take the form of a higher cost per transaction, or some other form.

That leaves registrars with a choice: either implement IDN support properly in
their domain management system and stores or don't offer IDNs.

K.

[1] I've yet to deal with the issue of variants all that well. That's
    something currently exercising my mind because until I come up with
    a solution, our support for CJK IDNs is less than ideal.

-- 
Keith Gaughan, Senior Developer
PGP/GPG key ID: 3E896381
Blacknight Internet Solutions Ltd. <http://blacknight.com/>
12A Barrowside Business Park, Carlow, Ireland
Registered in Ireland, Company No.: 370845
_______________________________________________
provreg mailing list
[email protected]
https://www.ietf.org/mailman/listinfo/provreg