Re: Unicode support (Was: Request for Comment: Callable CRM114 Classifiers (libcrm114))

   From: "Ger Hobbelt" <[email protected]>
   >
   > There is another story if libcrm114 will use some "external" encoding
   > (like UTF-8) *internally*. I.e. if libcrm114 will *not* accept wchar_t*
   > data, but rather char*, and require that this char* buffer be a UTF-8
   > stream. I do not mean to object it, it's up to you people who understand
   > how the data is used inside the classifiers, to decide. But still, I'd
   > like to point that most likely, any RE library (as well as libc
   > infrastructure: wcslen(), wcscmp() etc.) will require that the data is
   > passed in MBCS/wchar_t/"internal" representation. Because this is how
   > the unicode infrastructure is designed to work: always use wchar_t
   > inside C program, and only convert it from/to any suitable "external"
   > encoding (including but not limited to UTF-8) on input/output. So, if
   > you want to use MBCS support of the RE library, you'll need to convert
   > data to MBCS anyway.
   >
   > Whether MBCS support in the RE library is needed at all is a separate
   > question. It *is* needed if we want expressions like '/[aàá]bcd/'.
   > Do we want them or not - I do not know.

   Yup. It's a tough call. Your reply has made me think again. Thanks. ;-)

Same here.

   Things get a little hazy here because crm114 has two important
   sections when it comes to data processing:

   - the classifiers
   - the regex-assisted text processing facilities (in the crm114 script language).

   For the latter, it generally makes sense to assume we are processing
   'text' and that word means we've got to cope with code pages one way
   or another.

   The former (the classifiers) can also be said to work on 'text', but
   really they don't: they work on 'features', which take various forms:
   hashes, bits, ...

   Let's ditch the text processing/scripting section for a while when
   thinking about it and hence assume libcrm114 will only offer
   classifiers. (That'ld mean we might have a libcrmscript or simply the
   crm114 app itself if we want those text manipulating facilities too)

I already split the scripting language away for libcrm114.  No
scripting allowed.  


   Given that the classifiers don't need 'text' input to work -- they
   just need 'data' and a way to chop that 'data' into 'atoms' (called
   'features'; BitEntropy then uses 1-bit 'features') -- your reply has
   given me an idea that feels rather better than going for either UTF8
   or wchar_t: libcrm114 should not assume any 'encoding' AT ALL.

That's where we are now.

   In technical terms, that would mean libcrm114 classifier APIs should
   exhibit a mem*() like interface behaviour instead of a
   str*()/mbcstr*()/wstr*() one: void* pointers for data.

That's what the proposal I sent out does.  You pass a char *
(OK, maybe it should be tagged void *) of "classify-me data" 
and some control information which includes where to find the
statistics files, and it all flows from there.

   Plus callbacks, which can be configured by the libcrm114 user (=
   application developer) as [s]he will know what kind of data and
   possible encoding(s) will be fed to the classifiers: the only function
   block in any classifier that is concerned with the *format* of the
   data is the tokenizer (be it either a regex-assisted word extractor
   feeding the VT engine or a bit-chopper or ...): hence the way out is
   to have the tokenizer exist as an external (user defined) part, fed to
   the libcrm114 classifers with the data the tokenizer has to work on.

   As Bill pointed out, the hash algorithm may be adversely affected by
   wchar_t input. No problem there: the hash function is also part of the
   tokenizer, as classifiers are not interested in 'words' or
   'characters' (those are only 'intermediate data') as classifiers want
   to receive 'features'.

   A *classifier* doesn't care if the features it receives were conceived
   from RAW, MBCS, UTF-8 or wchar_t originals or whatnot; we munch
   'features' only, be it 32-/64-bit hashes or single bits.

Hmmm... an interesting idea.  Split the featurizer from the classifier.

That has some interesting implications.  Among them:

* Any tokenizer frontend can go on any classifier.  

* Users can create their own tokenizers that map non-textural data
  into featurespace (i.e. photos, power consumption graphs, stock
  market trends, SQL queries, .exe files can all get turned into
  "features")

HOWEVER - the way the data is packed is an issue.  Right now we
have several different packings:

* One bit, sequence important- used by the bit entropy classifier.
  This is identical to unaltered LATIN-1 / UTF-8 text.

* Eight bits, sequence important - used by the compression (LZ-77)
  classifier and the correllation classifier.  Also identical to
  unaltered LATIN-1 / UTF-8 text.

* 32 bits, sequence unimportant - used by Hyperspace, SVM, and Neural
  Network.  This is usually a single hash() of a feature.  Note that
  Hyperspace sorts these into ascending order for efficiency in
  processing.

* 64 bits, sequence unimportant - used by Markov, OSB, OSBF, and
  Winnow.  This is actually a pair of hash() values with different
  IVs and pipelines; the goal is to prevent birthday clashes.  We
  MIGHT want to obsolete this format and move the classifiers
  to 32-bit unsequenced features.

Note however that in some sense you can use these interchangeably;
one-bit and eight-bit are actually identical to LATIN-1 / UTF-8.
I *suspect* that if you feed a 32-bit unsorted ("as it comes from
the VT pipe") vector into bit entropy, it would work pretty well
too.  

OK- taken under advisement:

  int libcrm114_featureize ( char *input_text,
                             size_t input_len,
                             void *output_vector,
                             size_t output_len,
                             {VT params, whatever they are} );


   For ease of use of libcrm114 some 'often used tokenizers/feature
   hashers' can be provided with it, but that's an extra.

   Thanks for your remarks! They really helped clear the fog in my head.
   (I still wasn't happy with the Unicode/encoding stuff either way.)

yeah... me neither.

   This way, libcrm114 does not have to worry about encodings, language
   support or Unicode: those bits are implementation specific and should
   be provided from outside the library.
   (Take my own data analysis case, where I work on sampled data, which
   are numbers (signal levels really). *I* don't need nor want Unicode
   for my crm114 use, because it only adds a conversion layer that I will
   have to 'undo': twice the work for zero effect. The fact that most
   other folks will be munching email just means everyone else will just
   use a different 'tokenizer' than me; while all of us use the same
   libcrm114.)

Agreed.  Cleaner...

	 - Bill Yerazunis

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Crm114-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/crm114-general