Re: [Bitbucket] Issue #24: Missing glyphs from embedded emoji font (Symbola) (rptlab/reportlab)

On Tue, Feb 25, 2014 at 02:09:39PM +0000, Robin Becker wrote:
> to.textLine(u"Unicode symbols: \u02a4\U0001F631\U0001F64C\U0001F44C")
...
> I ran this on windows with python 3.3.3. The rsult shows the dz
> character \u02a4, but not the three astral plane emoji characters;
> they appear as ? chars.
> 
> I looked into the PDF produced by reportlab. We appear to be
> creating the subset map correctly at the end of the definitions I
> see this
> 
> <7F> <007F>
> <80> <02A4>
> <81> <1F631>
> <82> <1F64C>
> <83> <1F44C>
> 
> so we've seen those characters and allegedly created glyphs for
> them. In the body of the document I see this
> 
> (Unicode symbols: \200\201\202\203) Tj T* (UTF8 symbols: \200\201\202\203) Tj T* ET
> 
> So we're using the octal escapes for 0x80 0x81 0x82 0x83 in the
> string. From this I can only deduce that either we are failing in
> the glyph creation stage somewhere (ie when building the subset the
> glyph lookup fails) or we're building the subset correctly and
> Acrobat fails to deliver. I suspect the former.

That's very plausible.

> Debugging in Marius' ttfonts.py code reveals that we don't seem to
> read all of the glyphs. At line 641 our unichars lie in
> range(startCount[n],endCount[n]+1) and we are reading startCount &
> endCount with read_ushort() so all our unichars lie in 0<= unichar
> <= 0xffff.

I'm afraid it's been too long, and I don't remember the code.  Or the
TTF spec, for that matter.

I wouldn't be surprised to learn that characters outside the Basic
Multilingual Plane require special support.

> Seems there must be some kind of extension to let us read unicodes
> above 0xffff. We're using the first 'unicode' cmap table from
> 
> >cmap table 0/4: platFormID=0 encodingID=0 offset=00000024
> >cmap table 1/4: platFormID=1 encodingID=0 offset=00000144
> >cmap table 2/4: platFormID=3 encodingID=1 offset=0000034e
> >cmap table 3/4: platFormID=3 encodingID=10 offset=0000046e
> 
> and that appears to be a format 4 table which is what we read. Self
> evidently I'm missing something.

I think you're on the right track.

AFAIR the font files have multiple ways of mapping characters to glyph
numbers.  It may be a good idea to look at a recent version of the
TTF/OTF spec and see how non-BMP characters can be mapped to glyphs,
then fix ReportLab's font subsetting code to make sure it can find the
glyphs for these characters.

It sounds like a fun little exercise, but I'm afraid I won't be able to
find time to play with this any time soon.  :/

Marius Gedminas
-- 
The most effective debugging tool is still careful thought, coupled with
judiciously placed print statements.
        -- Brian W. Kernighan, in the paper Unix for Beginners (1979)

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.14 (GNU/Linux)

iD8DBQFTDRXbkVdEXeem148RAntEAJsF4L7b1QYbCBO/wCTpQu0VgMOA2gCfbD8B
DX643kAuQc5TH87A2jv8czE=
=eYle
-----END PGP SIGNATURE-----