Re: Patch to fix bug 134036

On Sunday 27 April 2008, Diego Iastrubni wrote:
>
> Yes, my implementation is still lacking. The next thing to be tested is
> the first char: does it force the paragraph to be LTR -> &LTE; ? or does
> it force it to be RTL -> &RTE;?
>
Oh, I see. The explicit marks. I suggest, then, that for this matter, you also 
consider a first implicit mark (RLM/LRM) as an explicit mark, because if 
someone bothered to put one in, fixing the direction is the only reasonable 
intent (unless I'm missing something) -- my opposition to the whole idea 
notwithstanding.

> Then... I wanted to save in the text those chars, only, and only, if the
> use inserted those manually into a katePart, or pressed control+shift on
> the corresponding side of the keyboard. bu this, IMHO, this can be used
> as an higher level protocol.
>

I agree -- that's context coming from outside the text.

> Quite frankly, I have been using this "first strong character" thing for
> several years, and I think I can do better. I have an SVN account, I
> hack ;-)
> Seriously, I was also thinking of putting this as a preference in
> KApplication, which direction auto-guessing protocol to use, but this
> will have to wait for KDE 4.2 as I cannot do this in time to 4.1 - it's
> frozen.
>

And I am still trying to convince you not to do that. For what does "do 
better" mean here? Correct me if I'm wrong, but my guess is "display more 
texts the way they were intended to be shown". However, how come we have 
texts that, when rendered according to the standard, are not displayed as 
intended? Simple: We have non-conformant implementations (and yes, that 
includes kmail without my patch -- its non-conformance comes about because 
its definition of paragraphs is inconsistent).

In fact, your alternative auto-guessing protocol is improving compatibility 
with non-conformant implementations at the expense of compatibility with 
conformant ones. For example, if I use gedit to write a line like

היום יצאו גרסאות חדשות ל-Ubuntu, Mandriva, PCLinuxOS, NetBSD ו-OpenOffice.org.

The new algorithm will render it LTR, because gedit wouldn't make me add any 
directionality marks. Such lines (and even paragraphs) are not as rare as one 
might think -- people include lists of Latin terms (companies, medications, 
you name it...) all the time.

Even out of Unicode, ISO-8859-8 and Windows-1255 have LRM and RLM (at 0xFD and 
0xFE). If you get text in one of these encodings, and the standard algorithm 
does not display it the way the creating application displayed it, then that 
application is non-standard-compliant -- be that standard Unicode or 
ISO-8859. 

So I ask again -- please don't let the bad guys win, by compromising on 
standard implementations.

Have fun,
	Shai.


> Shai Berger wrote:
> > (no longer cross-posting, as I think this is not an issue for kde-pim.
> > kde-core-devel, maybe).
> >
> > Hi Diego,
> >
> > On Saturday 26 April 2008, you wrote:
> >> Shai Berger wrote:
> >>> I think deciding to choose an algorithm that contradicts the Unicode
> >>> standard is a mistake -- you choose to present some texts differently
> >>> from standard-complying implementations. In the process, you kill the
> >>> option of putting an RLM/LRM as the first character to set the
> >>> direction, blur the distinction between strong and weak directionality,
> >>> and probably damage marks of direction embedding. Can you point me  to
> >>> the rationale behind this decision?
> >>
> >> I kindly disagree...
> >
> > You disagree that your implementation contradict the standard, that it
> > kills the paragraph-level effect of RLM/LRM, that it blurs the
> > distinctions, or that it damages embedding? I'll grant you the last two,
> > but the first two are valid (LRM/RLM should be obvious. see below about
> > contradicting the standard).
> >
> >> see seccion 4.3 of the Bidi Algorythm , called  Higher Levels protocols,
> >
> > Higher-level protocols are things like XML and word-processor file
> > formats, *not* plain text. Further, Section 4.3 ends with these words:
> >
> > 	When text using a higher-level protocol is to be converted to Unicode
> > plain text, for consistent appearance formatting codes should be inserted
> > to ensure that the order matches that of the higher-level protocol.
> >
> > So -- even if you consider KatePart to be implementing a higher-level
> > protocol, as long as it edits plain text, it should save it with the
> > correct marks. And this all but implies that it should be reading it
> > according to these marks as well.
> >
> >> and in seccion 3.3.1 we see this text (*):
> >>
> >> Because paragraph separators delimit text in this algorithm, this will
> >> *generally *be the first strong character after a paragraph separator or
> >> at the very beginning of the text. Note that the characters of type LRE,
> >> LRO, RLE, or RLO are ignored in this rule. This is because typically
> >> they are used to indicate that the embedded text is the /opposite/
> >> direction than the paragraph level.
> >
> > <argumentative>
> > Check your code, you forgot to ignore {LR,RL}{E,O}. Actually, you should
> > do more than ignore them: you should ignore anything between the {LR,RL}E
> > and the respective PDF, and count everything between the {LR,RL}O and its
> > PDF according to the explicit override (see 2.1, 2.2, 2.3).
> > </argumentative>
> >
> >> I think we can test using another algorythm for automatic determination
> >> of the text direction if the text itself does not include any LTR/RTL
> >> markups, nor the user specified any direction (input widgets). I just
> >> saw many ways in which the older code failed, and I want to test a new
> >> and "hopefuly better" way.
> >
> > I was going to refer you to 4.2 "conformance", but the argument really
> > boils down to understanding what "higher protocol" means. If I understand
> > you correctly, you think that any program displaying Unicode text -- in
> > particular, a plain-text editor -- is at liberty to define "higher level
> > protocols". I find this notion, with all due respect, absurd. In
> > particular, when applied to plain-text editors and plain-text mail, this
> > clearly breaks compatibility with other implementations of the standard.
> >
> > It is reasonable to apply the "higher protocol" notion to texts in an
> > application context (e.g, since you mentioned Akregator, the feed
> > language should determine the base direction of titles and articles; Meir
> > of Whatsup has been adamant about this for years). But note the important
> > difference: In cases like this, the context -- the "higher protocol" --
> > is *outside* the text whose direction we are trying to decide. Deciding
> > base direction only according to the text itself, while proclaiming the
> > right to do so (in a different way than the Unicode BiDi Algorithm) based
> > on the notion of a higher protocol -- I can only call this a fallacy. It
> > is wrong.
> >
> > Have fun,
> > 	Shai.



------------------------------------

Yahoo! Groups Links

<*> To visit your group on the web, go to:
    http://groups.yahoo.com/group/kde-il/

<*> Your email settings:
    Individual Email | Traditional

<*> To change settings online go to:
    http://groups.yahoo.com/group/kde-il/join
    (Yahoo! ID required)

<*> To change settings via email:
    mailto:[email protected] 
    mailto:[email protected]

<*> To unsubscribe from this group, send an email to:
    [email protected]

<*> Your use of Yahoo! Groups is subject to:
    http://docs.yahoo.com/info/terms/