utf8 character types ?

Kuniaki Mukai <[email protected]> Sat, 13 Sep 2014 00:01:47 +0900
Newsgroups gmane.comp.ai.prolog.swi
Message-ID <[email protected]>
Hi,

I have written a small script in swi to generate a dict named reserved_char_class
of  character types of swi-prolog  with characte intervals.  The last three
entries of the dict were added by hand for purpose to parse text in UTF8 encoding.

I have little knowledge about encoding like UTF8, but for 
testing regular expressions using the utf8  character classes,
the automata synthesized by my regular expression compiler recognises
input texts in Japanese charaters (Kanji, multi-bytes) as I expected. 

I hope this simplest addition of character classes for UTF8
will work without serious unexpected problems. If it works without problems, 
I, as a multi-bytes language user,  have to pay respects to inventors 
and developers of UTF8 encoding. 

reserved_char_class([
	 alnum-['0'-'9','A'-'Z',a-z],
	 alpha-['A'-'Z',a-z],
	 ascii-['\000\'-'\177\'],
	 cntrl-['\000\'-'\037\','\177\'-'\177\'],
	 csym-['0'-'9','A'-'Z','_'-'_',a-z],
	 csymf-['A'-'Z','_'-'_',a-z],
	 digit-['0'-'9'],
	 end_of_line-['\n'-'\r'],
	 graph-[ (!)- (~)],
	 lower-[a-z],
	 newline-['\n'-'\n'],
	 period-[ (!)- (!), ('.')- ('.'), (?)- (?)],
	 prolog_atom_start-[a-z],
	 prolog_identifier_continue-['0'-'9','A'-'Z','_'-'_',a-z],
	 prolog_symbol-[ (#)- ($), (&)- (&), (*)- (+), (-)- (/), (:)- (:), (<)- (@), (\)- (\), (^)- (^), (~)- (~)],
	 prolog_var_start-['A'-'Z','_'-'_'],
	 punct-[ (!)- (/), (:)- (@),'['- ('`'),'{'- (~)],
	 quote-['"'-'"','\''-'\'', ('`')- ('`')],
	 space-['\t'-'\r',' '-' '],
	 upper-['A'-'Z'],
	 white-['\t'-'\t',' '-' '],
	 utf8 -['\200\' - '\377\'],
	 utf8c -['\200\' - '\277\'],       %  UTF8 byte after the first one.
	 utf8b -['\300\' - '\377\']        %  the first byte of UTF8  codes.
		     
]).

Regards,

Kuniaki Mukai