Re: UTF-8 full support in LDIF / LDIF v2

On Jun 16, 2009, at 12:32 PM, Howard Chu wrote:

> Not to mention that to implement this properly will require complete  
> schema knowledge at the time the LDIF is generated. (Otherwise, how  
> do you distinguish a genuine octetString value, which cannot be  
> safely represented in UTF-8, from a directoryString value...)

Well, one could scan the value to see the octets are a valid UTF-8  
sequence of valid Unicode code points just as today most  
implementations scan data for octets within of SAFE-STRING.  The  
significant difference is the check is straight forward in LDIFv1, as  
it's san octet-by-octet check.  But if we allow UTF-8 sequences of  
valid Unicode points, each octet of the value must be checked to see  
that it's part of a valid UTF-8 sequence, and each UTF-8 sequence  
checked to see if encodes a valid Unicode code point.  And then  
wrapping becomes more complicated, etc..

And even with all of that, LDIF would still not be well-formed Unicode  
text.  And even if we solved that (by even more complex restrictions  
on what Unicode code point sequences can be represented as UTF-8  
instead of base64-encoded UTF-8), we'd have the problem of unintended  
Unicode transformations in transporting LDIF.  (We have this problem  
with LDIFv1, but it's generally limited to end-of-line characters.   
With UTF-8, data will be impacted.  For instance, consider MUAs (or  
the like) that might convert (on send or receive) text to Net-Unicode.)

I've expanded my UTF-8 LDIF with some more goofiness.




-- Kurt

_______________________________________________
Ldapext mailing list
[email protected]
https://www.ietf.org/mailman/listinfo/ldapext

version: 2

dn: cn=funky
bom:
smiley-face:☺
# only SPACE is special
no-break-space:  
zero-width-space:​
word-joiner:⁠
ideographic-space:　
zero-width-no-break-space:
# line separators and other such things
nel:…
ls:

ps:

ff:
# these hyphen differ but may look the same
hyphen-minus:-
hyphen:‐
non-breaking-hyphen:‑
figure-dash:‒
en-dash:–
minus-sign:−
roman-uncia-sign:𐆑
# these differ but may look the same
o-diaeresis:ö
o-diaeresis-decomposed:ö
# ignorables
ignore:⁠
ignore:⁡
ignore:­
# inside-out rule
inside-out:ạ̭̈̃
inside-out:ขึ้
# combining character
diaeresis:̈
# bidi
bidi:ڃ
bidi:ٱABٹڃ
bidi:ٱ37ٹڃ
bidi:‮ABC‬
bidi:‭ٱٹڃ‬
# bidi wrapped
bidi:
 ڃ
bidi:ٱA
 Bٹڃ
bidi:ٱ3
 7ٹڃ
bidi:‮A
 BC‬
bidi:‭ٱ
 ٹڃ‬
# private use
pu: