Re: UTF-8 full support in LDIF / LDIF v2

Thanks Kurt, sorry for my asking an answer in another email, it seems we 
were writing at the same time.


Kurt Zeilenga wrote:
>>
>> -the directory is broken
>> -you export to LDIF
>> -compare this LDIF with a previous one from when the directory was 
>> working.
> 
> You don't need UTF-8 for this.  A simple text diff tool will tell you 
> that the base64 differs.

True, diff will tell you that they are different, and where, but then you 
need to decode base64 to find what the text is in the two file to help you 
understand why they are different.

If there were no base64 encryption, then you would know right away, in most 
case, making it a much faster process.


> But now you assume you'll be able to read them.  This is a bad 
> assumption.

I still don't understand why you are saying this. Can you give precise 
examples ?

>  A simple diff tool might show two DIFFERENT values the same 
> way, leading the human to believe there is no difference when there is a 
> significant difference.

So, for example, one file contains U+2026 (ellipsis, "…") while the other 
contains three U_002E (three times the full stop character, "..."), and the 
issue of duplication in Unicode 
(http://en.wikipedia.org/wiki/Duplicate_characters_in_Unicode) ?

Well, what do we do today when diff shows us that there is a difference in 
two ascii files, but our eyes can't see it ? We hex dump of the offending 
line(s), and go "ahah, I've got <CR> here, but <CR><LF> there). On the other 
hand what's the percentage of time you diff files and run into this problem ?

Having those base64 encoded will make the fact that they are different more 
obvious, but won't help us when we're trying to understand why they are 
different, we will still have to go base64 --> UTF-8 --> hex values.

Data can be different because it comes from different sources, in which case 
it is likely that people will use different duplicate of Unicode point to 
express the same thing, but data can also be different because it was 
changed, which is the case I am more concerned with (backup of LDAP servers 
etc...), and in this case, I doubt somebody will change the mu symbol to the 
micro symbol or something similar. It might happen for malicious reasons, 
but I expect that to be a small percentage.


> 
> The key phrase here is "Unicode text".  And most such display tools not 
> only require "well-formed" text, but often cannot display all 
> "well-formed" text.  But removing the ASCII restriction does not make a 
> LDIF file "Unicode text".  It makes it a series of Unicode code points 
> and hence display of it as text will be quite problematic.  

I don't understand what you are saying here. Could you give an example of a 
problematic case, or a link to an explanation ?

Why is a series of Unicode point not text ?
What do you mean by "well-formed text", or actually by text that is not 
well-formed ?

> And even 
> it's displayable, you have the problem that two values might display in 
> the same way, making visual diff'ing problematic.

This is the issue above (Unicode duplication) ? Right ?


 >> Other case: People have mentioned scripts that build LDIF file from
 >> other source, and have mentioned that encoding the values in base64 is
 >> an overhead they could do without.
 >
 > While base64 data is an additional step, it's an additional step that
 > well supported today.  If we lift the ASCII restriction now, we'll have
 > some implementations that do support it and some that don't, and that
 > will cause interop problems.   I cannot support inducing such interop
 > problems without a strong justification.

Adding a new version of the standard, does not remove the old one. You can 
tell people that you only work with version 1 LDIF files. The advantage of 
having a standard, is that the tools will slowly adopt it and will be able 
to deal with it. If we don't create a standard, we run into the situation 
where everybody creates their own format, and create their own tools to 
transform their format to the current LDIF one. If I remember right, Jim 
mentioned that he already uses his version of extended LDIF.


-- 
Yves.
http://www.sollers.ca/

_______________________________________________
Ldapext mailing list
[email protected]
https://www.ietf.org/mailman/listinfo/ldapext