Re: Python 3 and pygments-parser
Marcus Harnisch <[email protected]> Wed, 29 May 2024 19:33:14 +0200
| Newsgroups | gmane.comp.gnu.global.bugs |
|---|---|
| Message-ID | <CAAmKruXRP52=D8m6hPWTgHorsmQY3Q7+vex40uRTdFrPCH2+YA@mail.gmail.com> |
--000000000000cfd37c06199b251a Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi Shigio I am thinking about tackling this feature in a reasonably useful and robust way. I am not concerned about Python 2.x, but wouldn't want to break compatibility either. As it stands, =E2=80=98latin1=E2=80=99 encoding is us= ed for implementing something like =E2=80=9Cbinary but with newlines=E2=80=9D. - The current implementation of pygments_parser.py is incomplete wrt I/O encoding and will probably break when challenged with characters outside the ASCII range. - Encodings of any form of input that are not ASCII-compatible are probably not going to work at all. - Many OS-facing functions, such as =E2=80=98os.getenv=E2=80=99, but als= o the low-level parts of =E2=80=98subprocess.Popen()=E2=80=99 use =E2=80=98sys.getfilesy= stemencoding()=E2=80=99 for determining the desired encoding. Most current unixoid OS are configured= to UTF-8 based locales, and even Python on Windows defaults to UTF-8 for OS-facing encoding (since 2016, Python 3.6+, PEP 529). - Any non-ASCII content of gtags.conf is most likely going to break pygments_parser.py in one way or another. I'd propose to rely on =E2=80=98sys.getfilesystemencoding()=E2=80=99 as well for reading. - Source code must be presented to Pygment's Lexers as string. Programming languages that allow non-ASCII source code would normally us= e UTF-8 (e.g. Python), which I'd recommend for =E2=80=98read_file()=E2=80= =99, possibly with an appropriate error handler. Depending on how a Lexer implements string handling, exotic encodings might even be less broken than before if byte= s are preserved via =E2=80=98surrogateescape=E2=80=99 or =E2=80=98backslas= hreplace=E2=80=99. IMHO, relying on the respective system default encoding in most places and an explicit UTF-8 in read_file() is going to improve compatibility and by side effect helps with unifying code paths between Python 2 and 3. Best regards, Marcus On Thu, May 16, 2024 at 12:42=E2=80=AFAM Marcus Harnisch < [email protected]> wrote: > Hi Shigio > > Glad to hear that it didn't work :-) Thank you for adding this to the > known bugs list. > > Best regards, > Marcus > > On Tue, May 14, 2024 at 8:16=E2=80=AFAM Shigio YAMAGUCHI <[email protected]>= wrote: > >> Hi Marcus, >> I confirmed that the problem is reproduced. >> I have made a new entry to the 'Known bugs' list. >> Thank you for the report. >> >> [https://www.gnu.org/software/global/bugs.html] >> o Pygments plug-in parser with python3 does not work, if 'ctagscom' is >> not set. >> If it is not set, default path obtained by configure script should be >> used. >> >> $ cat > gtags.conf >> default:\ >> :ctagscom=3D:\ >> :langmap=3DC\:.c.h:\ >> :gtags_parser=3DC\:/usr/local/lib/gtags/pygments-parser.la: >> $ gtags >> $ global -x '.*' >> $ _ # no tags >> >> Regards, >> Shigio >> >> On Mon, May 13, 2024 at 5:04=E2=80=AFPM Marcus Harnisch >> <[email protected]> wrote: >> > >> > Hi Shigio >> > >> > On Sat, May 11, 2024 at 5:35=E2=80=AFAM Shigio YAMAGUCHI <[email protected]= rg> >> wrote: >> >> >> >> $ cat gtags.conf >> >> default:\ >> >> :ctagscom=3D/opt/local/bin/uctags:\ >> >> :langmap=3DC\:.c.h:\ >> >> :gtags_parser=3DC\:/usr/local/lib/gtags/pygments-parser.la: >> > >> > >> > The important difference, which exposes the bug, is your explicit >> configuration of ctagscom. Leave it undefined and rely on whatever >> UNIVERSAL_CTAGS has been configured to. Only if ctagscom is empty, you w= ill >> see a comparison between b'' (empty bytearray) and '' (empty string). >> > >> > Best regards, >> > Marcus >> >> >> >> -- >> Shigio YAMAGUCHI <[email protected]> >> PGP fingerprint: >> 26F6 31B4 3D62 4A92 7E6F 1C33 969C 3BE3 89DD A6EB >> > --000000000000cfd37c06199b251a Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div class=3D"gmail_default" style=3D"font-size:small">Hi = Shigio</div><div class=3D"gmail_default" style=3D"font-size:small"><br></di= v><div class=3D"gmail_default" style=3D"font-size:small">I am thinking abou= t tackling this feature in a reasonably useful and robust way. I am not con= cerned about Python 2.x, but wouldn't want to break compatibility eithe= r. As it stands, =E2=80=98latin1=E2=80=99 encoding is used for implementing= something like =E2=80=9Cbinary but with newlines=E2=80=9D.</div><div class= =3D"gmail_default" style=3D"font-size:small"><ul><li>The current implementa= tion of pygments_parser.py is incomplete wrt I/O encoding and will probably= break when challenged with characters outside the ASCII range.</li><li>Enc= odings of any form of input that are not ASCII-compatible are probably not = going to work at all.<br></li><li>Many OS-facing functions, such as =E2=80= =98os.getenv=E2=80=99, but also the low-level parts of =E2=80=98subprocess.= Popen()=E2=80=99 use =E2=80=98sys.getfilesystemencoding()=E2=80=99 for dete= rmining the desired encoding. Most current unixoid OS are configured to UTF= -8 based locales, and even Python on Windows defaults to UTF-8 for OS-facin= g encoding (since 2016, Python 3.6+, PEP 529).</li><li>Any non-ASCII conten= t of gtags.conf is most likely going to break pygments_parser.py in one way= or another. I'd propose to rely on =E2=80=98sys.getfilesystemencoding(= )=E2=80=99 as well for reading.</li><li>Source code must be presented to Py= gment's Lexers as string. Programming languages that allow non-ASCII so= urce code would normally use UTF-8 (e.g. Python), which I'd recommend f= or =E2=80=98read_file()=E2=80=99, possibly with an appropriate error handle= r. Depending on how a Lexer implements string handling, exotic encodings mi= ght even be less broken than before if bytes are preserved via =E2=80=98sur= rogateescape=E2=80=99 or =E2=80=98backslashreplace=E2=80=99.<br></li></ul><= div>IMHO, relying on the respective system default encoding in most places = and an explicit UTF-8 in read_file() is going to improve compatibility and = by side effect helps with unifying code paths between Python 2 and 3.</div>= <div><br></div><div>Best regards,</div><div>Marcus<br></div></div></div><br= ><div class=3D"gmail_quote"><div dir=3D"ltr" class=3D"gmail_attr">On Thu, M= ay 16, 2024 at 12:42=E2=80=AFAM Marcus Harnisch <<a href=3D"mailto:marcu= [email protected]">[email protected]</a>> wrote:<br></div= ><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border= -left:1px solid rgb(204,204,204);padding-left:1ex"><div dir=3D"ltr"><div cl= ass=3D"gmail_default" style=3D"font-size:small">Hi Shigio</div><div><br></d= iv><div>Glad <span class=3D"gmail_default" style=3D"font-size:small">to hea= r that </span>it didn't work<span class=3D"gmail_default" style=3D"font= -size:small"> </span>:-)<span class=3D"gmail_default" style=3D"font-size:sm= all"> Thank you for adding this to the known bugs list.<br></span></div><di= v><span class=3D"gmail_default" style=3D"font-size:small"><br></span></div>= <div><span class=3D"gmail_default" style=3D"font-size:small">Best regards,<= /span></div><div><span class=3D"gmail_default" style=3D"font-size:small">Ma= rcus<br></span></div></div><br><div class=3D"gmail_quote"><div dir=3D"ltr" = class=3D"gmail_attr">On Tue, May 14, 2024 at 8:16=E2=80=AFAM Shigio YAMAGUC= HI <<a href=3D"mailto:[email protected]" target=3D"_blank">[email protected]</= a>> wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"margin:0p= x 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi= Marcus,<br> I confirmed that the problem is reproduced.<br> I have made a new entry to the 'Known bugs' list.<br> Thank you for the report.<br> <br> [<a href=3D"https://www.gnu.org/software/global/bugs.html" rel=3D"noreferre= r" target=3D"_blank">https://www.gnu.org/software/global/bugs.html</a>]<br> o Pygments plug-in parser with python3 does not work, if 'ctagscom'= is not set.<br> =C2=A0 If it is not set, default path obtained by configure script should b= e used.<br> <br> $ cat > gtags.conf<br> default:\<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 :ctagscom=3D:\<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 :langmap=3DC\:.c.h:\<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 :gtags_parser=3DC\:/usr/local/lib/gtags/<a href= =3D"http://pygments-parser.la" rel=3D"noreferrer" target=3D"_blank">pygment= s-parser.la</a>:<br> $ gtags<br> $ global -x '.*'<br> $ _=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0# no tags<br> <br> Regards,<br> Shigio<br> <br> On Mon, May 13, 2024 at 5:04=E2=80=AFPM Marcus Harnisch<br> <<a href=3D"mailto:[email protected]" target=3D"_blank">marcus= [email protected]</a>> wrote:<br> ><br> > Hi Shigio<br> ><br> > On Sat, May 11, 2024 at 5:35=E2=80=AFAM Shigio YAMAGUCHI <<a href= =3D"mailto:[email protected]" target=3D"_blank">[email protected]</a>> wrote:<= br> >><br> >> $ cat gtags.conf<br> >> default:\<br> >> :ctagscom=3D/opt/local/bin/uctags:\<br> >> :langmap=3DC\:.c.h:\<br> >> :gtags_parser=3DC\:/usr/local/lib/gtags/<a href=3D"http://pygments= -parser.la" rel=3D"noreferrer" target=3D"_blank">pygments-parser.la</a>:<br= > ><br> ><br> > The important difference, which exposes the bug, is your explicit conf= iguration of ctagscom. Leave it undefined and rely on whatever UNIVERSAL_CT= AGS has been configured to. Only if ctagscom is empty, you will see a compa= rison between b'' (empty bytearray) and '' (empty string).<= br> ><br> > Best regards,<br> > Marcus<br> <br> <br> <br> -- <br> Shigio YAMAGUCHI <<a href=3D"mailto:[email protected]" target=3D"_blank">sh= [email protected]</a>><br> PGP fingerprint:<br> 26F6 31B4 3D62 4A92 7E6F=C2=A0 1C33 969C 3BE3 89DD A6EB<br> </blockquote></div> </blockquote></div> --000000000000cfd37c06199b251a--