utf8 character types ?
Kuniaki Mukai <[email protected]> Sat, 13 Sep 2014 00:01:47 +0900
| Newsgroups | gmane.comp.ai.prolog.swi |
|---|---|
| Message-ID | <[email protected]> |
Hi,
I have written a small script in swi to generate a dict named reserved_char_class
of character types of swi-prolog with characte intervals. The last three
entries of the dict were added by hand for purpose to parse text in UTF8 encoding.
I have little knowledge about encoding like UTF8, but for
testing regular expressions using the utf8 character classes,
the automata synthesized by my regular expression compiler recognises
input texts in Japanese charaters (Kanji, multi-bytes) as I expected.
I hope this simplest addition of character classes for UTF8
will work without serious unexpected problems. If it works without problems,
I, as a multi-bytes language user, have to pay respects to inventors
and developers of UTF8 encoding.
reserved_char_class([
alnum-['0'-'9','A'-'Z',a-z],
alpha-['A'-'Z',a-z],
ascii-['\000\'-'\177\'],
cntrl-['\000\'-'\037\','\177\'-'\177\'],
csym-['0'-'9','A'-'Z','_'-'_',a-z],
csymf-['A'-'Z','_'-'_',a-z],
digit-['0'-'9'],
end_of_line-['\n'-'\r'],
graph-[ (!)- (~)],
lower-[a-z],
newline-['\n'-'\n'],
period-[ (!)- (!), ('.')- ('.'), (?)- (?)],
prolog_atom_start-[a-z],
prolog_identifier_continue-['0'-'9','A'-'Z','_'-'_',a-z],
prolog_symbol-[ (#)- ($), (&)- (&), (*)- (+), (-)- (/), (:)- (:), (<)- (@), (\)- (\), (^)- (^), (~)- (~)],
prolog_var_start-['A'-'Z','_'-'_'],
punct-[ (!)- (/), (:)- (@),'['- ('`'),'{'- (~)],
quote-['"'-'"','\''-'\'', ('`')- ('`')],
space-['\t'-'\r',' '-' '],
upper-['A'-'Z'],
white-['\t'-'\t',' '-' '],
utf8 -['\200\' - '\377\'],
utf8c -['\200\' - '\277\'], % UTF8 byte after the first one.
utf8b -['\300\' - '\377\'] % the first byte of UTF8 codes.
]).
Regards,
Kuniaki Mukai