Re: utf-8 output in a terminal
Grégory Vanuxem <[email protected]> Mon, 29 Jul 2024 10:51:38 +0200
| Newsgroups | gmane.lisp.openmcl.devel |
|---|---|
| Message-ID | <CAHnU2dY+iZxjz_vpszOKXyMwEMa0kggNBmODg8AZUhH8rroApQ@mail.gmail.com> |
--000000000000ca5a6a061e5ef8c1 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Oh, many thanks, I prefer your solution ;) Greg Le lun. 29 juil. 2024 =C3=A0 04:14, R. Matthew Emerson <[email protected]> a = =C3=A9crit : > > > > > On Jul 28, 2024, at 6:31=E2=80=AFPM, Gr=C3=A9gory Vanuxem <g.vanuxem@gm= ail.com> wrote: > > > > Found! > > > > In level-0/l0-misc.lisp there is exactly what i was looking for. > > (ccl::%get-utf-8-cstring ..) It's a pity I think it is not documented > > since now UTF-8 is relatively widely used. > > > > All is right now! > > > The exported functionality is ccl:get-encoded-string. > > (get-encoded-string :utf-8 cname (#_strlen cname)) > > https://ccl.clozure.com/docs/ccl.html#f_get-encoded-string > > The manual also contains examples of using both ccl:with-encoded-cstrs and ccl:get-encoded-string: > > https://ccl.clozure.com/docs/ccl.html#brief-examples > > > > > > > Thanks > > > > Greg > > > > Le lun. 29 juil. 2024 =C3=A0 01:34, Gr=C3=A9gory Vanuxem <g.vanuxem@gma= il.com> a =C3=A9crit : > >> > >> Hello Matthew, > >> > >> Le dim. 28 juil. 2024 =C3=A0 07:31, R. Matthew Emerson <[email protected]= m> a =C3=A9crit : > >>> > >>> > >>> > >>>> On Jul 27, 2024, at 8:23=E2=80=AFPM, Gr=C3=A9gory Vanuxem <g.vanuxem= @gmail.com> wrote: > >>>> > >>>> In FriCAS built on top of Clozure CL I obtain this output in a terminal (WSL2) > >>>> > >>>> (gamma(x) =C3=83=C3=82=C2=A2=C3=83=C3=82=C3=83=C3=82 reals && Re(x) = > 0) || (x =C3=83=C3=82=C2=A2=C3=83=C3=82=C3=83=C3=82 integers && > >>>> gamma(x) =C3=83=C3=82=C2=A2=C3=83=C3=82=C3=83=C3=82 reals) > >>>> > >>>> instead of (SBCL) > >>>> > >>>> (gamma(x) =E2=88=88 reals && Re(x) > 0) || (x =E2=88=89 integers && = gamma(x) =E2=88=88 reals) > >>>> > >>>> if I (print ...) a string with, for example, =E2=88=88, in a pure Cl= ozure CL > >>>> (recently git cloned) this is correctly printed, no problem. > >>>> > >>>> So I wonder if I have to modify the output stream character encoding > >>>> or the output routine. > >>> > >>> Check that the external format of the FriCAS output stream is what you expect. In other words, if you pass an explicit :external-format argument to cl:open, make sure it matches what your terminal requires. > >> > >> I don't think it is at this stage in fact, I was wrong. Better > >> checking what is going on, since the string comes from C (sorry I > >> forgot to mention that) I tend to think my problem comes from > >> ccl::%get-cstring. Tracing it seems to show it's not "decoding" it and > >> considers it as a 8859-1 string. But to test this i.e. force CCL to > >> use :utf8 format to "read it" I have not found. There is no option > >> from what I know to get-cstring like :external-format and using > >> with-encoded-cstrs does not seem to work. > >> > >> Right now I am using some FFI macros from FriCAS to simplify my code, > >> I will try to revert to pure CCL code first to check my claims, > >> TIMTOWTDI. > >> > >>> Doing (describe *terminal-io*) might show something interesting. Here=E2=80=99s what it shows for me: > >>> > >>> ? (describe *terminal-io*) > >>> #<ECHOING-TWO-WAY-STREAM input #<BASIC-CHARACTER-INPUT-STREAM UTF-8 (TTY/0) #x1824C7D6>, output #<BASIC-CHARACTER-OUTPUT-STREAM UTF-8 (TTY/1) #x1824C2A6> #x1825FBE6> > >>> Class: #<STANDARD-CLASS ECHOING-TWO-WAY-STREAM> > >>> Wrapper: #<CLASS-WRAPPER ECHOING-TWO-WAY-STREAM #x18039A16> > >>> Instance slots > >>> SHARED-RESOURCE: NIL > >>> OPEN-P: T > >>> INPUT-STREAM: #<BASIC-CHARACTER-INPUT-STREAM UTF-8 (TTY/0) #x1824C7D6= > > >>> OUTPUT-STREAM: #<BASIC-CHARACTER-OUTPUT-STREAM UTF-8 (TTY/1) #x1824C2A6> > >>> > >>> The default external format has been #<EXTERNAL-FORMAT :UTF-8/:UNIX #x18249C6E> for quite a long time now. > >>> > >>> https://ccl.clozure.com/docs/ccl.html#characters-and-external-formats > >> > >> This is exactly what I have when I use this function in FriCAS. I have > >> read the documentation and the internal utf32 representation but here > >> I am stuck. This is not the most important thing I need to do but it > >> will be imperative for some next stuff. > >> > >> In anyway, thanks! > >> > >> Greg > >> > >>> > >>>> > >>>> And more importantly, how can I achieve that? > >>>> > >>>> - Greg > >>>> > >>>> PS: > >>>> (1) -> )lisp CCL:*DEFAULT-EXTERNAL-FORMAT* > >>>> Value =3D :UNIX > >>> > --000000000000ca5a6a061e5ef8c1 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div>Oh, many thanks, I prefer your solution ;)<br> <br> Greg</div><div><br> <br> Le lun. 29 juil. 2024 =C3=A0 04:14, R. Matthew Emerson <<a href=3D"mailt= o:[email protected]" target=3D"_blank">[email protected]</a>> a =C3=A9crit := <br> ><br> ><br> ><br> > > On Jul 28, 2024, at 6:31=E2=80=AFPM, Gr=C3=A9gory Vanuxem <<a = href=3D"mailto:[email protected]" target=3D"_blank">[email protected]</= a>> wrote:<br> > ><br> > > Found!<br> > ><br> > > In level-0/l0-misc.lisp there is exactly what i was looking for.<= br> > > (ccl::%get-utf-8-cstring ..) It's a pity I think it is not do= cumented<br> > > since now UTF-8 is relatively widely used.<br> > ><br> > > All is right now!<br> ><br> ><br> > The exported functionality is ccl:get-encoded-string.<br> ><br> > (get-encoded-string :utf-8 cname (#_strlen cname))<br> ><br> > <a href=3D"https://ccl.clozure.com/docs/ccl.html#f_get-encoded-string"= rel=3D"noreferrer" target=3D"_blank">https://ccl.clozure.com/docs/ccl.html= #f_get-encoded-string</a><br> ><br> > The manual also contains examples of using both ccl:with-encoded-cstrs= and ccl:get-encoded-string:<br> ><br> > <a href=3D"https://ccl.clozure.com/docs/ccl.html#brief-examples" rel= =3D"noreferrer" target=3D"_blank">https://ccl.clozure.com/docs/ccl.html#bri= ef-examples</a><br> ><br> ><br> ><br> > ><br> > > Thanks<br> > ><br> > > Greg<br> > ><br> > > Le lun. 29 juil. 2024 =C3=A0 01:34, Gr=C3=A9gory Vanuxem <<a h= ref=3D"mailto:[email protected]" target=3D"_blank">[email protected]</a= >> a =C3=A9crit :<br> > >><br> > >> Hello Matthew,<br> > >><br> > >> Le dim. 28 juil. 2024 =C3=A0 07:31, R. Matthew Emerson <<a= href=3D"mailto:[email protected]" target=3D"_blank">[email protected]</a>> = a =C3=A9crit :<br> > >>><br> > >>><br> > >>><br> > >>>> On Jul 27, 2024, at 8:23=E2=80=AFPM, Gr=C3=A9gory Van= uxem <<a href=3D"mailto:[email protected]" target=3D"_blank">g.vanuxem= @gmail.com</a>> wrote:<br> > >>>><br> > >>>> In FriCAS=C2=A0 built on top of Clozure CL I obtain t= his output in a terminal (WSL2)<br> > >>>><br> > >>>> (gamma(x) =C3=83=C3=82=C2=A2=C3=83=C3=82=C3=83=C3=82 = reals && Re(x) > 0) || (x =C3=83=C3=82=C2=A2=C3=83=C3=82=C3=83= =C3=82 integers &&<br> > >>>> gamma(x) =C3=83=C3=82=C2=A2=C3=83=C3=82=C3=83=C3=82 r= eals)<br> > >>>><br> > >>>> instead of (SBCL)<br> > >>>><br> > >>>> (gamma(x) =E2=88=88 reals && Re(x) > 0) ||= (x =E2=88=89 integers && gamma(x) =E2=88=88 reals)<br> > >>>><br> > >>>> if I (print ...) a string with, for example, =E2=88= =88, in a pure Clozure CL<br> > >>>> (recently git cloned) this is correctly printed, no p= roblem.<br> > >>>><br> > >>>> So I wonder if I have to modify the output stream cha= racter encoding<br> > >>>> or the output routine.<br> > >>><br> > >>> Check that the external format of the FriCAS output strea= m is what you expect. In other words, if you pass an explicit :external-for= mat argument to cl:open, make sure it matches what your terminal requires.<= br> > >><br> > >> I don't think it is at this stage in fact, I was wrong. B= etter<br> > >> checking what is going on, since the string comes from C (sor= ry I<br> > >> forgot to mention that) I tend to think my problem comes from= <br> > >> ccl::%get-cstring. Tracing it seems to show it's not &quo= t;decoding" it and<br> > >> considers it as a 8859-1 string. But to test this i.e. force = CCL to<br> > >> use :utf8 format to "read it" I have not found. The= re is no option<br> > >> from what I know to get-cstring like :external-format and usi= ng<br> > >> with-encoded-cstrs does not seem to work.<br> > >><br> > >> Right now I am using some FFI macros from FriCAS to simplify = my code,<br> > >> I will try to revert to pure CCL code first to check my claim= s,<br> > >> TIMTOWTDI.<br> > >><br> > >>> Doing (describe *terminal-io*) might show something inter= esting. Here=E2=80=99s what it shows for me:<br> > >>><br> > >>> ? (describe *terminal-io*)<br> > >>> #<ECHOING-TWO-WAY-STREAM input #<BASIC-CHARACTER-IN= PUT-STREAM UTF-8 (TTY/0) #x1824C7D6>, output #<BASIC-CHARACTER-OUTPUT= -STREAM UTF-8 (TTY/1) #x1824C2A6> #x1825FBE6><br> > >>> Class: #<STANDARD-CLASS ECHOING-TWO-WAY-STREAM><br> > >>> Wrapper: #<CLASS-WRAPPER ECHOING-TWO-WAY-STREAM #x1803= 9A16><br> > >>> Instance slots<br> > >>> SHARED-RESOURCE: NIL<br> > >>> OPEN-P: T<br> > >>> INPUT-STREAM: #<BASIC-CHARACTER-INPUT-STREAM UTF-8 (TT= Y/0) #x1824C7D6><br> > >>> OUTPUT-STREAM: #<BASIC-CHARACTER-OUTPUT-STREAM UTF-8 (= TTY/1) #x1824C2A6><br> > >>><br> > >>> The default external format has been #<EXTERNAL-FORMAT= :UTF-8/:UNIX #x18249C6E> for quite a long time now.<br> > >>><br> > >>> <a href=3D"https://ccl.clozure.com/docs/ccl.html#characte= rs-and-external-formats" rel=3D"noreferrer" target=3D"_blank">https://ccl.c= lozure.com/docs/ccl.html#characters-and-external-formats</a><br> > >><br> > >> This is exactly what I have when I use this function in FriCA= S. I have<br> > >> read the documentation and the internal utf32 representation = but here<br> > >> I am stuck. This is not the most important thing I need to do= but it<br> > >> will be imperative for some next stuff.<br> > >><br> > >> In anyway, thanks!<br> > >><br> > >> Greg<br> > >><br> > >>><br> > >>>><br> > >>>> And more importantly, how can I achieve that?<br> > >>>><br> > >>>> - Greg<br> > >>>><br> > >>>> PS:<br> > >>>> (1) -> )lisp CCL:*DEFAULT-EXTERNAL-FORMAT*<br> > >>>> Value =3D :UNIX<br> > >>><br> ><br> </div> --000000000000ca5a6a061e5ef8c1--