Re: utf8 character types ?

Hi,

you didn't say what your reserved_char_class/1 is supposed to do, so I
cannot really comment on it. But you really want to read this article:

http://kunststube.net/encoding/

In particular, make sure that you understand the difference between a
character set (such as Unicode) and a character encoding (such as UTF-8).

Best,
Kilian

On 09/12/2014 05:01 PM, Kuniaki Mukai wrote:
> Hi,
> 
> I have written a small script in swi to generate a dict named reserved_char_class
> of  character types of swi-prolog  with characte intervals.  The last three
> entries of the dict were added by hand for purpose to parse text in UTF8 encoding.
> 
> I have little knowledge about encoding like UTF8, but for 
> testing regular expressions using the utf8  character classes,
> the automata synthesized by my regular expression compiler recognises
> input texts in Japanese charaters (Kanji, multi-bytes) as I expected. 
> 
> I hope this simplest addition of character classes for UTF8
> will work without serious unexpected problems. If it works without problems, 
> I, as a multi-bytes language user,  have to pay respects to inventors 
> and developers of UTF8 encoding. 
> 
> reserved_char_class([
> 	 alnum-['0'-'9','A'-'Z',a-z],
> 	 alpha-['A'-'Z',a-z],
> 	 ascii-['\000\'-'\177\'],
> 	 cntrl-['\000\'-'\037\','\177\'-'\177\'],
> 	 csym-['0'-'9','A'-'Z','_'-'_',a-z],
> 	 csymf-['A'-'Z','_'-'_',a-z],
> 	 digit-['0'-'9'],
> 	 end_of_line-['\n'-'\r'],
> 	 graph-[ (!)- (~)],
> 	 lower-[a-z],
> 	 newline-['\n'-'\n'],
> 	 period-[ (!)- (!), ('.')- ('.'), (?)- (?)],
> 	 prolog_atom_start-[a-z],
> 	 prolog_identifier_continue-['0'-'9','A'-'Z','_'-'_',a-z],
> 	 prolog_symbol-[ (#)- ($), (&)- (&), (*)- (+), (-)- (/), (:)- (:), (<)- (@), (\)- (\), (^)- (^), (~)- (~)],
> 	 prolog_var_start-['A'-'Z','_'-'_'],
> 	 punct-[ (!)- (/), (:)- (@),'['- ('`'),'{'- (~)],
> 	 quote-['"'-'"','\''-'\'', ('`')- ('`')],
> 	 space-['\t'-'\r',' '-' '],
> 	 upper-['A'-'Z'],
> 	 white-['\t'-'\t',' '-' '],
> 	 utf8 -['\200\' - '\377\'],
> 	 utf8c -['\200\' - '\277\'],       %  UTF8 byte after the first one.
> 	 utf8b -['\300\' - '\377\']        %  the first byte of UTF8  codes.
> 		     
> ]).
> 
> Regards,
> 
> Kuniaki Mukai
> 
> _______________________________________________
> SWI-Prolog mailing list
> [email protected]
> https://lists.iai.uni-bonn.de/mailman/listinfo.cgi/swi-prolog
>