Character encodings issue with phrase_from_file

Too long didn't read:  My input file has ascii \223 and \224.  If I take
them out, phrase_from_file succeeds. If I leave them in, it fails.  What
can I do?

I've been working on a  parser for the USDA nutritional database.  It's
basically a '^' delimited DOS text file.  However, it uses some strange
characters \223 and \224 (strange characters, I know).  These seem to
defeat Prolog. The term that is defeated is "stri" seen below

%%%%%%%%%%%% My Code %%%%%%%%%%%%%%%%

:- use_module(library(dcg/basics)).
:- use_module(library(pio)).
:- encoding(utf8).

sep --> "^".
delim --> "~".

code(Atom, L,_) --> delim,!, digits(D), delim, { length(D, L),
atom_codes(Atom, D) }.
code([],_,false) --> [].
stri(Str, L,_) --> delim,!, string(C), delim, { length(C, I), I =< L,
string_to_list(Str, C) }.
stri([],_,false) --> [].
inti(N, L,_) -->  digit(D0),!, digits(D), { length(D,DL), DL + 1 =< L,
number_codes(N, [D0|D]) }.
inti([],_,false) --> [].
floa(N, LW,LP,_) --> digit(W0),!, digits(W), ".", !, digit(P0), digits(P),
{ length(W, WL),WL + 1 =< LW,
  length(P, PL),PL + 1 =< LP,
  append([[W0|W], ".", [P0|P]], DIGITS),
  number_codes(N, DIGITS)
}.
floa([],_,_,false) --> [].

food_description(fdrecord(NDB_ID, FDGRP_CODE, LONG_DESC, SHORT_DESC,
COMM_NAME, MANUFAC_NAME, SURVEY, REF_DESC, REFUSE, SCI_NAME, N_FACTOR,
PRO_FACTOR, FAT_FACTOR, CHO_FACTOR)) -->
    code(NDB_ID,       5,   true),  sep,
    code(FDGRP_CODE,   4,   true),  sep,
    stri(LONG_DESC,    200, true),  sep,
    stri(SHORT_DESC,   60,  true),  sep,
    stri(COMM_NAME,    100, false), sep,
    stri(MANUFAC_NAME, 100, false), sep,
    stri(SURVEY,       1,   false), sep,
    stri(REF_DESC,     135, false), sep,
    inti(REFUSE,       2,   false), sep,
    stri(SCI_NAME,     65,  false), sep,
    floa(N_FACTOR,     4,2, false), sep,
    floa(PRO_FACTOR,   4,2, false), sep,
    floa(FAT_FACTOR,   4,2, false), sep,
    floa(CHO_FACTOR,   4,2, false), ("\n"; "\r\n"),
    { format('~w|~w|~w|~w|~w|~w|~w|~w|~w|~w|~w|~w|~w|~w~n', [NDB_ID,
FDGRP_CODE, LONG_DESC, SHORT_DESC, COMM_NAME, MANUFAC_NAME, SURVEY,
REF_DESC, REFUSE, SCI_NAME, N_FACTOR, PRO_FACTOR, FAT_FACTOR, CHO_FACTOR ])
}.

food_descriptions([T|TN]) --> food_description(T),!, food_descriptions(TN).
food_descriptions([]) --> [].


%%%%%%% Sample Problem Input %%%%%%%%%%%%

Problem Input (the last record GATORADE G performance O 2, the blue
characters are ascii codes).

~14390~^~1400~^~Cocoa mix, with aspartame, powder, prepared with
water~^~COCOA MIX,W/ ASPRT,PDR,PREP W/ H2O~^~hot
chocolate~^~~^~~^~~^0^~~^^^^
~14440~^~1400~^~Water, bottled, non-carbonated, DANNON Fluoride To
Go~^~WATER,BTLD,NON-CARBONATED,DANNON FLUORIDE TO GO~^~~^~~^~~^~~^0^~~^^^^
~14450~^~1400~^~Drink mix, QUAKER OATS, GATORADE, orange flavor,
powder~^~DRINK MIX,QUAKER OATS,GATORADE,ORANGE FLAVOR,PDR~^~~^~Quaker Oats
Company - The Gatorade Company,  a unit of Pepsi Co.~^~~^~~^0^~~^^^^
~14460~^~1400~^~Sports drink, PEPSICO QUAKER GATORADE, GATORADE, original,
fruit-flavored, ready-to-drink. Now called \223“G performance \224O
2”.~^~SPORTS DRK,PEPSICO QUAKER GATORADE,ORIGINAL,FRUIT-FLAVD,RTD~^~thirst
quencher, Gatorade G02 Perform~^~Quaker Oats Company - The Gatorade
Company,  a unit of Pepsi Co.~^~Y~^~~^0^~~^^^^

(or you can download the full database:
http://www.ars.usda.gov/Services/docs.htm?docid=8964)

%%%%%%%%% How It's Run %%%%%%%%%%%%%%%%

The input is in a DOS formatted text file (I called this subset
FOOD_DES_SMALL.txt). I load it with:

phrase_from_file(food_descriptions(WHAT), 'db/FOOD_DES_SMALL.txt').


Cheers,
Warren Wilkinson
1 403 383-7152 <http://www.formlis.com>
-------------- next part --------------
HTML attachment scrubbed and removed
_______________________________________________
SWI-Prolog mailing list
[email protected]
https://lists.iai.uni-bonn.de/mailman/listinfo.cgi/swi-prolog