[ xdvi-Bugs-2902167 ] Bug? Incorrect rendering of combining characters in Hebrew

Bugs item #2902167, was opened at 2009-11-22 21:52
Message generated for change (Comment added) made by stefanulrich
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=377580&aid=2902167&group_id=23164

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: fonts
Group: incorrect behaviour
>Status: Closed
>Resolution: Wont Fix
Priority: 5
Private: No
Submitted By: Technion (technioniit)
>Assigned to: Stefan Ulrich (stefanulrich)
Summary: Bug?  Incorrect rendering of combining characters in Hebrew

Initial Comment:
Bug?  Incorrect rendering of combining characters in Hebrew

This is complicated; I'm not even sure if it's a bug in xdvik.
(This is my first bug report in my entire life, and I'm not a professional developer, so apologies in advance.)

Using Ivritex + Culmus-latex + ucs packages, Hebrew nequddot (vowel points) don't get rendered properly as combining characters when the .dvi file is displayed in xdvi; however, the subsequent PostScript file generated from the .dvi file using dvips typesets the vowel points correctly (as does subsequent conversion of the resulting PostScript file to PDF, so it seems that the .dvi file is okay).

This bug report is probably going to be useless as there are just too many packages and LaTeX .sty files involved.

Uploaded files in xdvi.tar:
  - ivritex.tex:  LaTeX demo input file (but this also invokes a number of other files)
  - ivrit.tex:  Template file used by ivritex.tex (invokes several LaTeX/TeX packages)
  - ivritex.dvi:  .dvi file created by latex (MacTeX-2009, 20091108 release)
  - ivritex_xdvi.tiff:  Screenshot of above dvi file as rendered by xdvi from MacTeX-2009 running under Apple Mac OS X 10.5.8 Leopard; note that vowel points INCORRECTLY get rendered following (to the left of) the corresponding base characters, rather than directly underneath/above, as if there were no support for combining characters
  - ivritex.ps:  PostScript file generated from the above .dvi file; here the vowel points are typeset correctly
  - ivritex.pdf:  PDF file generated from above PostScript file, with vowel points rendered correctly

NOTE:  Had to omit ivritex.ps from xdvi.tar above; PS file is ~ 600 kB; maximum size of files uploaded as part of bug reports = 256 kB.
However, the PDF file above was generated directly from the PostScript file (which itself was generated directly from the .dvi file)

Using MacTeX-2009, release of 20091108.

xdvi version:  xdvik version 22.84.16 (Xaw toolkit); Libraries: kpathsea version 5.0.0, T1lib version 5.1.2
uname -a:  Darwin mac.local 9.8.0 Darwin Kernel Version 9.8.0: Wed Jul 15 16:57:01 PDT 2009; root:xnu-1228.15.4~1/RELEASE_PPC Power Macintosh
X -version:  X.org Release 7.3; X.Org X Server 1.4.2-apple33; Build Date: 20090617

xdvi -debug shows that the correct fonts with nekkudot (vowel points) are getting loaded


----------------------------------------------------------------------

>Comment By: Stefan Ulrich (stefanulrich)
Date: 2009-11-24 22:54

Message:
Hi,

I'm glad that you found the reason for this issue! Yes indeed, Type3 fonts
are not supported by T1lib (xdvik ought to switch to freetype at some point
...), so I'll close this bug as won't fix.

----------------------------------------------------------------------

Comment By: Technion (technioniit)
Date: 2009-11-24 10:40

Message:
Hi Stefan,

You are amazing.  You identified two elements (the _nikud_ package and
_t1lib_ [I don't even know how you knew about the _nikud_ package {or even
knew that I was invoking it, since it didn't appear in any of the files I
uploaded}]) that turn out to be precisely the explanation for the problems
I was having.  You totally hit it on the nail!

After checking out these two, as you suggested, it turns out that, as you
somehow suspected, the vowel-point-placing mechanism for Hebrew text with
nequddot (vowel points) in Ivritex is handled by the _nikud_ package.  It
appears that the _nikud_ package implements Hebrew vowel point placing
using Type 3 PostScript fonts (that add the vowel point placing mechanism
as arbitrary PostScript procedures, which are permitted in Type 3 fonts)
that wrap standard Type 1 Culmus fonts (which provide the glyphs and even
some of the vowel points, but don't provide the vowel-point-placing
mechanisms).

I did as you said and tried using the -not1lib option, and voila, almost
all of the vowel points disappeared.

>From the above I conclude that what is going on is this:  The
vowel-point-placing mechanisms are provided as part of the Type 3 fonts
provided by the _nikud_ package.  _t1lib_ apparently doesn't support full
Type 3 fonts (i. e., doesn't support the full PostScript language within
the fonts), and therefore cannot process the _nikud_ package's Type 3 font
wrappers around the Type 1 Culmus fonts; as a result, the
vowel-point-placing mechanisms implemented as full PostScript procedures in
the Type 3 fonts simply never get invoked by _xdvik_ + _t1lib_.  In
contrast, when processing the .dvi file using _dvips_, this isn't a
problem, since _dvips_ apparently supports Type 3 fonts.

The entire mechanism is explained in detail in
http://www.cs.tau.ac.il/%7Estoledo/Pubs/vowels.ps which is linked from the
_nikud_ project page at http://nikud.berlios.de/

So in spite of your extremely humble and self-deprecating comment that
this could be a bug with _xdvik_, it turns out that I was totally wrong;
this has nothing to do with _xdvik_; the problem arises because _t1lib_
doesn't support Type 3 fonts (i. e., the entire PostScript language, rather
than the subset permitted for Type 1 fonts), which isn't their fault either
because the whole point of their project was to support Type 1 fonts (they
never claimed to support Type 3 fonts), hence the name of their library.

So unless _t1lib_ eventually adds full support for Type 3 fonts (which was
never their mission), the only solution I can see is to use a PostScript
previewer when previewing documents with Hebrew vowel points implemented
using the _nikud_ package.

Sorry for all the trouble.  I should've suspected that, after all these
years, _xdvik_ is probably already nearly as perfect and bug-free as it can
possibly be.  If you ever come to Israel to visit, your first beer is on me
:-)

P. S.  If out of curiosity you want to try reproducing the entire LaTeX
`package stack' I was using, see
http://www.technion.ac.il/~ak/linux/hebrew.html (note that this is for a
Linux environment, although the results were the same for both teTeX
running under Linux and the MacTeX-2009 distribution running under Apple
Mac OS X Version 10.5.8 `Leopard').

----------------------------------------------------------------------

Comment By: Stefan Ulrich (stefanulrich)
Date: 2009-11-23 21:34

Message:
Hi,

this is really difficult to reproduce since it requires all those packages
and fonts (tried to install the Nikud package, but I didn't manage to get
the fonts installed properly). It could be a problem with the font metrics
or a bug in xdvik. Is the result the same when you use "xdvi -not1lib
ivritex"?

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=377580&aid=2902167&group_id=23164

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day 
trial. Simplify your report design, integration and deployment - and focus on 
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july