Re: Character encodings issue with phrase_from_file

On 18/08/2013, at 12:27 PM, Warren Wilkinson wrote:

> Too long didn't read:  My input file has ascii \223 and \224.  If I take
> them out, phrase_from_file succeeds. If I leave them in, it fails.  What
> can I do?

Are we talking about the characters
U+0093 SET TRANSMIT STATE  (= octal 223)
U_0094 CANCEL CHARACTER    (= octal 224)
or
U+00DF LATIN SMALL LETTER SHARP S (= decimal 223)
U+00E0 LATIN SMALL LETTER A WITH GRAVE (= decimal 224)
here?

The web page you pointed to says the "SR25 Data Files" are
in "ASCII".  Whatever
\223 and \224 are, they are *definitely* not ASCII.

If those numbers are in octal, they are not, strictly speaking,
ISO Latin 1 either.  If this comes from DOS originally, it could
be in some weird code page such as CP437.

Looking at the line displayed as
	Now called \223“G performance \224O<newline>2”
it seems likely to me that those characters are intended to be
some sort of quotation marks.  Quite a lot of Windows code pages
have \223 (octal) = LEFT DOUBLE QUOTATION MARK and \224 (octal) =
RIGHT DOUBLE QUOTATION MARK; all of CP1250..CP1258 do so.

There's a long shameful history of web sites providing information
in some Windows or DOS code page and claiming it is ASCII or Latin 1.

What I would do on a Unix box is

	iconv -f CP1250 -t UTF-8 FOOD_DES.TXT >FOOD_DES.UTF

and similarly for the other files.

If you have Cygwin, you have iconv, and there is a native port
win_iconv.exe.

If SWI Prolog supports CP1250 as encoding, you can just use that.
Me, I'd convert the files, so that I didn't have to _keep_
remembering to specify an encoding when opening them.