Re: Character encodings issue with phrase_from_file

When I was scraping e-commerce sites, I stumbled upon some unknown (to me)
characters: among others there were

maps(223, [0'“]).
maps(224, [0'”]).

that SWI-Prolog let me place in source. Then I *presume* you could extend
your grammar accepting that data.

openq --> [0'“].
closeq --> [0'”].
testg --> openq, "hello", closeq.

 1 ?- phrase(testg, "“hello”").

true.





2013/8/19 Richard A. O'Keefe <[email protected]>

>
> On 18/08/2013, at 12:27 PM, Warren Wilkinson wrote:
>
> > Too long didn't read:  My input file has ascii \223 and \224.  If I take
> > them out, phrase_from_file succeeds. If I leave them in, it fails.  What
> > can I do?
>
> Are we talking about the characters
> U+0093 SET TRANSMIT STATE  (= octal 223)
> U_0094 CANCEL CHARACTER    (= octal 224)
> or
> U+00DF LATIN SMALL LETTER SHARP S (= decimal 223)
> U+00E0 LATIN SMALL LETTER A WITH GRAVE (= decimal 224)
> here?
>
> The web page you pointed to says the "SR25 Data Files" are
> in "ASCII".  Whatever
> \223 and \224 are, they are *definitely* not ASCII.
>
> If those numbers are in octal, they are not, strictly speaking,
> ISO Latin 1 either.  If this comes from DOS originally, it could
> be in some weird code page such as CP437.
>
> Looking at the line displayed as
>         Now called \223“G performance \224O<newline>2”
> it seems likely to me that those characters are intended to be
> some sort of quotation marks.  Quite a lot of Windows code pages
> have \223 (octal) = LEFT DOUBLE QUOTATION MARK and \224 (octal) =
> RIGHT DOUBLE QUOTATION MARK; all of CP1250..CP1258 do so.
>
> There's a long shameful history of web sites providing information
> in some Windows or DOS code page and claiming it is ASCII or Latin 1.
>
> What I would do on a Unix box is
>
>         iconv -f CP1250 -t UTF-8 FOOD_DES.TXT >FOOD_DES.UTF
>
> and similarly for the other files.
>
> If you have Cygwin, you have iconv, and there is a native port
> win_iconv.exe.
>
> If SWI Prolog supports CP1250 as encoding, you can just use that.
> Me, I'd convert the files, so that I didn't have to _keep_
> remembering to specify an encoding when opening them.
>
>
> _______________________________________________
> SWI-Prolog mailing list
> [email protected]
> https://lists.iai.uni-bonn.de/mailman/listinfo.cgi/swi-prolog
>
-------------- next part --------------
HTML attachment scrubbed and removed