Bug#959474: Issues with Chinese language (all variants) when building some pages in buster

On Mon, May 04, 2020 at 10:19:02PM -0400, Boyuan Yang wrote:
> Mwei (https://nm.debian.org/person/mwei/) just talked to me saying that it
> could be a bug with isSPACE_L1 macro in perl's pp.c. He will be replying the
> email soon.
> 

Hi,

(I used reportbug to handle reply of this thread, and I missed a lot of
recipients here.  This is a resend of reply in #959474.  Sorry for the
noise.)

After a bit of investigation of Perl source code (5.31.11 downloaded
from upstream) I found the they have weird handling of whitespace when
`feature unicode_strings` turned on.  I am not a perl person and I
haven't executed the source code yet, so my interpretation might be
wrong.

When `unicode_strings` is on, `in_uni_8_bit` should true internally, and
in three places of pp.c:6040, pp.c:6076, pp.c:6114 `isSPACE_L1` is
called to check whether the examining character is a whitespace, by
checking whether the character is 0x85 or 0xA0 (handy.h:1611).  In the
case of the character 包, the last byte of 3-byte UTF-8 code is 0x85,
henceforth the problem.

-----BEGIN PGP SIGNATURE-----

iQIzBAABCgAdFiEE8htgli/cvAeDZWXRl1ewRNNETDMFAl6w2hcACgkQl1ewRNNE
TDOjMQ/+Iymr8trng5ePfmM27z86VKJa+YJAojmpkbjdiAsfzIV6WzfhGScJqhqj
Cm43uwgC4jMoMWf1GUWqZMYi54hRMvN6Fc0O8yt6Pacnl24P6SiJ5GpHuoBSoMeC
kGz6QkTy3ZINgrDKtGsnSurTnJn/Q4WvIH4nNBl8uyPH8Mt1MOpVcZgJE0Gg7mkT
Zvc616q3DoX8UgAs7cFKzrNz4Nl/MDuphTibMZ8CBC4mVrUq6VTfgZEem4F0014p
mXf2OZKAVOAxVMPDnwudLBMWZGLOZ8Jq2NsYdHE9xwYmc+HgdTLEMMQjzMYqmx2M
lBsCzHEXNVA8Z6nCXSQzPq0ndsaYtcTUBwkeh4XkLB4jvb1aSTFwWhn9pgOwDzBm
SRiz2JSRp6oqYuiAXQuOxkj50DjhXlflnHFXCN46tKq6RmmxnbujQbN1yxfchZWy
uO0EbUGbxz/iIGFwekwZD4VfZRg7U7oCGeXJKgYFu/7/uQtO209i+q0SScS1u+JJ
SZM6mBA6HQXCIIhgNtSEXJxK3b6fVrAFD2AGuTjqSmGcZxcSLAy/NKRfyXmOj/fR
LvFcUx7movtQt9EHFKv0GeT3oCr27k8z95ApH1KLQ6iUsXcR6CQzcAoajnHjcewI
kew+Me24FS7dvJ5EnfrkAZmLZNTI+m4R8vWaSrWYi3wDi8Vflqw=
=eQHi
-----END PGP SIGNATURE-----