Re: Unicode support (Was: Request for Comment: Callable CRM114 Classifiers (libcrm114))

Ger Hobbelt wrote:

>> In mail classification, there *has* to be multiple charset support.
> 
> I know. I am only saying those encoding should be handled outside of
> CRM114 - as you seem to say as well, though you seem to expect it in
> the 'wrapping application' with which you /might/ mean crm114?

In fact, I meant "all that is above libcrm114" which may be a pipe of
some re-encoder and the crm114 binary. I have no problem if the crm114
binary only support one specific encoding (provided that it covers all
languages). Although *if* external encoding (like UTF-8) is *not* going
to be the same as internal (like, MBCS aka wchar_t aka mb*() RTL
functions), I see no difference in implementing one specific encoding
vs. any encoding supported by system locale.

There is another story if libcrm114 will use some "external" encoding
(like UTF-8) *internally*. I.e. if libcrm114 will *not* accept wchar_t*
data, but rather char*, and require that this char* buffer be a UTF-8
stream. I do not mean to object it, it's up to you people who understand
how the data is used inside the classifiers, to decide. But still, I'd
like to point that most likely, any RE library (as well as libc
infrastructure: wcslen(), wcscmp() etc.) will require that the data is
passed in MBCS/wchar_t/"internal" representation. Because this is how
the unicode infrastructure is designed to work: always use wchar_t
inside C program, and only convert it from/to any suitable "external"
encoding (including but not limited to UTF-8) on input/output. So, if
you want to use MBCS support of the RE library, you'll need to convert
data to MBCS anyway.

Whether MBCS support in the RE library is needed at all is a separate
question. It *is* needed if we want expressions like '/[aàá]bcd/'.
Do we want them or not - I do not know.

Eugene

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/

_______________________________________________
Crm114-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/crm114-general

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFI21+Jfrw/cIw6UWkRAnI7AJ9KKMptr0h4akxeHv3/wX80lac4vgCgsGTO
GFERGn6ep9KjN/0QjuHc9pc=
=g6Oe
-----END PGP SIGNATURE-----