Re: Python 3 and pygments-parser

Hi Marcus,

Are you talking about a bug or a feature addition?
If it is a bug, could you please explain the specific steps to reproduce it=
?
If it is a new feature, could you please explain the specification?
Thank you in advance.

Regards,
Shigio

On Thu, May 30, 2024 at 2:33=E2=80=AFAM Marcus Harnisch
<[email protected]> wrote:
>
> Hi Shigio
>
> I am thinking about tackling this feature in a reasonably useful and robu=
st way. I am not concerned about Python 2.x, but wouldn't want to break com=
patibility either. As it stands, =E2=80=98latin1=E2=80=99 encoding is used =
for implementing something like =E2=80=9Cbinary but with newlines=E2=80=9D.
>
> The current implementation of pygments_parser.py is incomplete wrt I/O en=
coding and will probably break when challenged with characters outside the =
ASCII range.
> Encodings of any form of input that are not ASCII-compatible are probably=
 not going to work at all.
> Many OS-facing functions, such as =E2=80=98os.getenv=E2=80=99, but also t=
he low-level parts of =E2=80=98subprocess.Popen()=E2=80=99 use =E2=80=98sys=
.getfilesystemencoding()=E2=80=99 for determining the desired encoding. Mos=
t current unixoid OS are configured to UTF-8 based locales, and even Python=
 on Windows defaults to UTF-8 for OS-facing encoding (since 2016, Python 3.=
6+, PEP 529).
> Any non-ASCII content of gtags.conf is most likely going to break pygment=
s_parser.py in one way or another. I'd propose to rely on =E2=80=98sys.getf=
ilesystemencoding()=E2=80=99 as well for reading.
> Source code must be presented to Pygment's Lexers as string. Programming =
languages that allow non-ASCII source code would normally use UTF-8 (e.g. P=
ython), which I'd recommend for =E2=80=98read_file()=E2=80=99, possibly wit=
h an appropriate error handler. Depending on how a Lexer implements string =
handling, exotic encodings might even be less broken than before if bytes a=
re preserved via =E2=80=98surrogateescape=E2=80=99 or =E2=80=98backslashrep=
lace=E2=80=99.
>
> IMHO, relying on the respective system default encoding in most places an=
d an explicit UTF-8 in read_file() is going to improve compatibility and by=
 side effect helps with unifying code paths between Python 2 and 3.
>
> Best regards,
> Marcus
>
> On Thu, May 16, 2024 at 12:42=E2=80=AFAM Marcus Harnisch <marcus.harnisch=
@verilab.com> wrote:
>>
>> Hi Shigio
>>
>> Glad to hear that it didn't work :-) Thank you for adding this to the kn=
own bugs list.
>>
>> Best regards,
>> Marcus
>>
>> On Tue, May 14, 2024 at 8:16=E2=80=AFAM Shigio YAMAGUCHI <[email protected]=
> wrote:
>>>
>>> Hi Marcus,
>>> I confirmed that the problem is reproduced.
>>> I have made a new entry to the 'Known bugs' list.
>>> Thank you for the report.
>>>
>>> [https://www.gnu.org/software/global/bugs.html]
>>> o Pygments plug-in parser with python3 does not work, if 'ctagscom' is =
not set.
>>>   If it is not set, default path obtained by configure script should be=
 used.
>>>
>>> $ cat > gtags.conf
>>> default:\
>>>         :ctagscom=3D:\
>>>         :langmap=3DC\:.c.h:\
>>>         :gtags_parser=3DC\:/usr/local/lib/gtags/pygments-parser.la:
>>> $ gtags
>>> $ global -x '.*'
>>> $ _                             # no tags
>>>
>>> Regards,
>>> Shigio
>>>
>>> On Mon, May 13, 2024 at 5:04=E2=80=AFPM Marcus Harnisch
>>> <[email protected]> wrote:
>>> >
>>> > Hi Shigio
>>> >
>>> > On Sat, May 11, 2024 at 5:35=E2=80=AFAM Shigio YAMAGUCHI <shigio@gnu.=
org> wrote:
>>> >>
>>> >> $ cat gtags.conf
>>> >> default:\
>>> >> :ctagscom=3D/opt/local/bin/uctags:\
>>> >> :langmap=3DC\:.c.h:\
>>> >> :gtags_parser=3DC\:/usr/local/lib/gtags/pygments-parser.la:
>>> >
>>> >
>>> > The important difference, which exposes the bug, is your explicit con=
figuration of ctagscom. Leave it undefined and rely on whatever UNIVERSAL_C=
TAGS has been configured to. Only if ctagscom is empty, you will see a comp=
arison between b'' (empty bytearray) and '' (empty string).
>>> >
>>> > Best regards,
>>> > Marcus
>>>
>>>
>>>
>>> --
>>> Shigio YAMAGUCHI <[email protected]>
>>> PGP fingerprint:
>>> 26F6 31B4 3D62 4A92 7E6F  1C33 969C 3BE3 89DD A6EB



--=20
Shigio YAMAGUCHI <[email protected]>
PGP fingerprint:
26F6 31B4 3D62 4A92 7E6F  1C33 969C 3BE3 89DD A6EB