Surprising clash of encoders with pikepdf (with patch)

Lennart Regebro via reportlab-users <[email protected]> Fri, 7 Jan 2022 18:16:00 +0100

Newsgroups	gmane.comp.python.reportlab.user
Message-ID	<CAHT-kBf2n2c2uhgEjvahwAqTkz=a+zRU3mvUzocHh9=umfpvrg@mail.gmail.com>

Hi all!

Both Reportlab and PikePDF registers "pdfdoc" encodings, which means that
which encoding you actually end up using is arbitrary. I guess it depends
on the import order, but I haven't checked.

That's all and well in itself, and shouldn't be a problem, but alas,
PikePDF's encoding is using the qpdf library, and that library will not
tell you which character failed to encode. Therefore, it doesn't raise
UnicodeEncodeError which requires that information, but ValueError. This is
actually specified in the docs for encode() and decode():

"encoding errors raise ValueError
<https://docs.python.org/3/library/exceptions.html#ValueError> (or a more
codec specific subclass, such as UnicodeEncodeError
<https://docs.python.org/3/library/exceptions.html#UnicodeEncodeError>)" -
https://docs.python.org/3/library/codecs.html

In other places it says "Raise UnicodeError
<https://docs.python.org/3/library/exceptions.html#UnicodeError> (or a
subclass); this is the default. Implemented in strict_errors()
<https://docs.python.org/3/library/codecs.html#codecs.strict_errors>." -
https://docs.python.org/3/library/codecs.html#error-handlers

I made a PR to change pikepdf's error from ValueError to UnicodeError which
has been iomplemented, but that only fixes half the problem. I believe
Reportlab should make one or both of these minor changes:

1. Catch UnicodeErrorsinstead of UnicodeEncodeError when "pdfdoc" encoding
is used. This should have no drawbacks.

2. When it uses the pdfdoc codec it should use it directly, and not via the
"pdfdoc" name.

The first fix is trivial so I didn't do that, but I attach a patch for the
second fix here.
I hope attachemnts works for this.

pdfdoc.diff (text/x-patch, 3.5 KB)

diff --git a/src/reportlab/pdfbase/pdfdoc.py b/src/reportlab/pdfbase/pdfdoc.py
index 91827d5..d40f7c2 100755
--- a/src/reportlab/pdfbase/pdfdoc.py
+++ b/src/reportlab/pdfbase/pdfdoc.py
@@ -16,7 +16,7 @@ classes are made available elsewhere for users to manipulate.
 """
 import types, binascii, codecs, time
 from collections import OrderedDict
-from reportlab.pdfbase import pdfutils
+from reportlab.pdfbase import pdfutils, rl_codecs
 from reportlab import rl_config
 from reportlab.lib.utils import import_zlib, open_for_read, makeFileName, isSeq, isBytes, isUnicode, _digester, isStr, bytestr, annotateException, TimeStamp
 from reportlab.lib.rl_accel import escapePDF, fp_str, asciiBase85Encode, asciiBase85Decode
@@ -27,6 +27,8 @@ from sys import platform
 from sys import version_info
 from sys import stderr
 
+pdfdoc_encoding = rl_codecs.RL_Codecs._rl_codecs('pdfdoc')
+
 if platform[:4] == 'java' and version_info[:2] == (2, 1):
     # workaround for list()-bug in Jython 2.1 (should be fixed in 2.2)
     def list(sequence):
@@ -208,7 +210,7 @@ class PDFDocument(PDFObject):
             filename = getattr(f,'name',None)
             if isinstance(filename,int):
                 filename = '<os fd:%d>'% filename
-            elif not isStr(filename): #try to fix bug reported by Robert Schroll <rschroll at gmail.com> 
+            elif not isStr(filename): #try to fix bug reported by Robert Schroll <rschroll at gmail.com>
                 filename = '<%s@0X%8.8X>' % (f.__class__.__name__,id(f))
             filename = makeFileName(filename)
         elif isStr(filename):
@@ -216,7 +218,7 @@ class PDFDocument(PDFObject):
             filename = makeFileName(filename)
             f = open(filename, "wb")
         else:
-            raise TypeError('Cannot use %s as a filename or file' % repr(filename)) 
+            raise TypeError('Cannot use %s as a filename or file' % repr(filename))
 
         data = self.GetPDFData(canvas)
         if isUnicode(data):
@@ -574,7 +576,7 @@ def _isbalanced(s):
 def _checkPdfdoc(utext):
     '''return true if no Pdfdoc encoding errors'''
     try:
-        utext.encode('pdfdoc')
+        pdfdoc_encoding.encode(utext)
         return 1
     except UnicodeEncodeError as e:
         return 0
@@ -610,19 +612,19 @@ class PDFString(PDFObject):
                     else:
                         u = s.decode('utf8')
                     if _checkPdfdoc(u):
-                        s = u.encode('pdfdoc')
+                        s = pdfdoc_encoding.encode(u)[0]
                     else:
                         s = codecs.BOM_UTF16_BE+u.encode('utf_16_be')
                 except:
                     try:
-                        s.decode('pdfdoc')
+                        pdfdoc_encoding.decode(s)[0]
                     except:
                         stderr.write('Error in %s' % (repr(s),))
                         raise
         elif isUnicode(s):
             if enc == 'auto':
                 if _checkPdfdoc(s):
-                    s = s.encode('pdfdoc')
+                    s = pdfdoc_encoding.encode(s)[0]
                 else:
                     s = codecs.BOM_UTF16_BE+s.encode('utf_16_be')
             elif self.unicodeEncValid:
@@ -1711,7 +1713,7 @@ class TextAnnotation(HighlightAnnotation):
                 Rect,
                 Contents,
                 QuadPoints=kw.pop("QuadPoints",None) or rect_to_quad(Rect),
-                Color=kw.pop("Color",(0,0,0)), 
+                Color=kw.pop("Color",(0,0,0)),
                 **kw)
     def Dict(self):
         d = HighlightAnnotation.Dict(self)