Re: Character encodings issue with phrase_from_file

On 19/08/2013, at 5:32 PM, Carlo Capelli wrote:

> When I was scraping e-commerce sites, I stumbled upon some unknown (to me) characters: among others there were
> 
> maps(223, [0'“]).
> maps(224, [0'”]).

There seems to be a mistake here.
OCTAL 223 and 224 represent double quotes in several Windows code pages.
DECIMAL 223 and 224 represent nothing in particular (upper half block,
that kind of thing).
> 
> that SWI-Prolog let me place in source.

The problem is that SWI Prolog is by default processing the external
file in the expectation that it is coded using UTF-8, and 
a bare \223 or \224 (a) isn't legal at all in UTF8 and
(b) doesn't represent a quotation mark in Unicode.

> Then I *presume* you could extend your grammar accepting that data.
> 
> openq --> [0'“].
> closeq --> [0'”].

It isn't a problem with DCGs or strings or for that matter Prolog.
The problem is that the file is being *treated as* UTF-8 when in
fact it is almost certainly some Windows 8-bit code page, most
like CP1250.

In fact the quotation marks occur only inside text fields that
the original poster's program shouldn't need to look inside.

There are basically only two things that will work.
- Recode the files using iconv, or
- Tell SWI Prolog to use the cp1250 encoding for the file.

According to the on-line manual,
"SWI-Prolog currently defines and supports the following encodings:
octet, ascii, iso_latin_1, text, utf8, unicode_be, unicode_le."

At least for reading, it would probably be a good thing if it
also supported cp_1250.

Jan, I'll send you a private message with the file from the Unicode
mappings directory.