Re: non-ASCII filepaths in a C function
David Turner <[email protected]> Sat, 25 Jul 2015 22:04:54 +0100
| Newsgroups | gmane.comp.lang.haskell.libraries,gmane.comp.lang.haskell.ffi |
|---|---|
| Message-ID | <CAErCyQ-ggad7UY2yXsZ5yj-GR1oxjYfF7P6a-hMLrd9s7r7xjw@mail.gmail.com> |
--===============7669358599340896648== Content-Type: multipart/alternative; boundary=001a113f9be4e1579f051bb978a6 --001a113f9be4e1579f051bb978a6 Content-Type: text/plain; charset=UTF-8 Hi, The native representation for filepaths on Linux is char[] (i.e. raw bytes). withCString converts from String to char[] using the current locale, which doesn't always work (at least, it doesn't always do what you want). As long as everything is in the same locale, ideally UTF-8, then you'll be fine, but it's legitimate to have a file whose name is not legal UTF-8 even in a UTF-8 locale, and these will cause you problems. (Minor, nitpicky bugbear: the native representation for filepaths on Windows is wchar_t[] which is interpreted as UTF-16 *where possible*, but there are also some legal filenames (e.g. "C:\\Temp\\\xd800") which are invalid as UTF-16) I'm not familiar with soxlib specifically, but for opening a file on Windows named as a char[] I'm going to guess that the library ultimately ends up calling a so-called ANSI version of a function like CreateFileA, which accepts a char[] and converts it to wchar_t[] within the OS according to the current code page. withCString seems to look at the current code page when converting a String to a char[] too, but clearly something's not matching for you. So a few things to check: - does soxlib use the ANSI version, CreateFileA or similar? - what code page does it think it's in? - can you convert the troublesome filename to bytes in this code page by hand, and compare with what withCString is doing? - can you convert these bytes to wchar_t[] using MultiByteToWideChar in the current code page? Does this look like what you expect? Unfortunately there's no complete general solution to this problem that fits through an API that only uses char[] for filenames - the mapping from filenames written as char[] to Windows filenames is never surjective. The best solution would be for soxlib to offer an API that accepted wchar_t[] filenames on Windows, although I appreciate this might not be reasonable! Hopefully this helps a bit. On 25 July 2015 at 08:40, Malcolm Wallace <[email protected]> wrote: > I believe the native representation for FilePaths on Windows should be > UTF16 strings. > > Regards, > Malcolm > > > On 24 Jul 2015, at 22:52, Henning Thielemann < > [email protected]> wrote: > > > > > > In my 'soxlib' package I have written a binding to > > > > sox_format_t * sox_open_read( > > char const * path, > > sox_signalinfo_t const * signal, > > sox_encodinginfo_t const * encoding, > > char const * filetype); > > > > > > I construct the C filepath "path" from a Haskell FilePath using > Foreign.C.String.withCString. This works for ASCII and non-ASCII characters > in Linux. However, non-ASCII characters let sox_open_read fail on Windows. > What is the correct way to convert FilePath to "char *"? > > _______________________________________________ > > Libraries mailing list > > [email protected] > > http://mail.haskell.org/cgi-bin/mailman/listinfo/libraries > _______________________________________________ > Libraries mailing list > [email protected] > http://mail.haskell.org/cgi-bin/mailman/listinfo/libraries > --001a113f9be4e1579f051bb978a6 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr">Hi,<div><br></div><div>The native representation for filep= aths on Linux is char[] (i.e. raw bytes). withCString converts from String = to char[] using the current locale, which doesn't always work (at least= , it doesn't always do what you want). As long as everything is in the = same locale, ideally UTF-8, then you'll be fine, but it's legitimat= e to have a file whose name is not legal UTF-8 even in a UTF-8 locale, and = these will cause you problems.</div><div><br></div><div>(Minor, nitpicky bu= gbear: the native representation for filepaths on Windows is wchar_t[] whic= h is interpreted as UTF-16 *where possible*, but there are also some legal = filenames (e.g. "C:\\Temp\\\xd800") which are invalid as UTF-16)<= br></div><div><br></div><div>I'm not familiar with soxlib specifically,= but for opening a file on Windows named as a char[] I'm going to guess= that the library ultimately ends up calling a so-called ANSI version of a = function like CreateFileA, which accepts a char[] and converts it to wchar_= t[] within the OS according to the current code page. withCString seems to = look at the current code page when converting a String to a char[] too, but= clearly something's not matching for you.</div><div><br></div><div>So = a few things to check:</div><div><br></div><div>- does soxlib use the ANSI = version, CreateFileA or similar?</div><div>- what code page does it think i= t's in?</div><div>- can you convert the troublesome filename to bytes i= n this code page by hand, and compare with what withCString is doing?</div>= <div>- can you convert these bytes to wchar_t[] using=C2=A0<span style=3D"c= olor:rgb(0,0,0);font-family:Consolas,Courier,monospace;font-size:13px;line-= height:17.5499992370605px">MultiByteToWideChar</span>=C2=A0in the current c= ode page? Does this look like what you expect?</div><div><br></div><div>Unf= ortunately there's no complete general solution to this problem that fi= ts through an API that only uses char[] for filenames - the mapping from fi= lenames written as char[] to Windows filenames is never surjective. The bes= t solution would be for soxlib to offer an API that accepted wchar_t[] file= names on Windows, although I appreciate this might not be reasonable!</div>= <div><br></div><div>Hopefully this helps a bit.</div></div><div class=3D"gm= ail_extra"><br><div class=3D"gmail_quote">On 25 July 2015 at 08:40, Malcolm= Wallace <span dir=3D"ltr"><<a href=3D"mailto:[email protected]" ta= rget=3D"_blank">[email protected]</a>></span> wrote:<br><blockquote= class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc soli= d;padding-left:1ex">I believe the native representation for FilePaths on Wi= ndows should be UTF16 strings.<br> <br> Regards,<br> =C2=A0 =C2=A0 Malcolm<br> <span class=3D""><br> > On 24 Jul 2015, at 22:52, Henning Thielemann <<a href=3D"mailto:lem= [email protected]">[email protected]</a>> wrote:<br= > ><br> ><br> > In my 'soxlib' package I have written a binding to<br> ><br> > sox_format_t * sox_open_read(<br> >=C2=A0 =C2=A0 char=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0const * path,<br> >=C2=A0 =C2=A0 sox_signalinfo_t=C2=A0 =C2=A0const * signal,<br> >=C2=A0 =C2=A0 sox_encodinginfo_t const * encoding,<br> >=C2=A0 =C2=A0 char=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0const * filetype);<br> ><br> ><br> > I construct the C filepath "path" from a Haskell FilePath us= ing Foreign.C.String.withCString. This works for ASCII and non-ASCII charac= ters in Linux. However, non-ASCII characters let sox_open_read fail on Wind= ows. What is the correct way to convert FilePath to "char *"?<br> > _______________________________________________<br> > Libraries mailing list<br> </span>> <a href=3D"mailto:[email protected]">[email protected]<= /a><br> > <a href=3D"http://mail.haskell.org/cgi-bin/mailman/listinfo/libraries"= rel=3D"noreferrer" target=3D"_blank">http://mail.haskell.org/cgi-bin/mailm= an/listinfo/libraries</a><br> _______________________________________________<br> Libraries mailing list<br> <a href=3D"mailto:[email protected]">[email protected]</a><br> <a href=3D"http://mail.haskell.org/cgi-bin/mailman/listinfo/libraries" rel= =3D"noreferrer" target=3D"_blank">http://mail.haskell.org/cgi-bin/mailman/l= istinfo/libraries</a><br> </blockquote></div><br></div> --001a113f9be4e1579f051bb978a6-- --===============7669358599340896648== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Libraries mailing list [email protected] http://mail.haskell.org/cgi-bin/mailman/listinfo/libraries --===============7669358599340896648==--