Re: Unicode support (Was: Request for Comment: Callable CRM114 Classifiers (libcrm114))

On Thu, Sep 25, 2008 at 4:35 PM, Bill Yerazunis <[email protected]> wrote:
> I already split the scripting language away for libcrm114.  No
> scripting allowed.

Yup. Hadn't really 'seen' that (I speedread the document and read the
words but they never made it past the eyes. :-((  Happens to me
sometimes.)

> That's what the proposal I sent out does.  You pass a char *
> (OK, maybe it should be tagged void *) of "classify-me data"
> and some control information which includes where to find the
> statistics files, and it all flows from there.

I was going for 'void*' in the upcoming comments on the RFC, yes. ;-)


> Hmmm... an interesting idea.  Split the featurizer from the classifier.
>
> That has some interesting implications.  Among them:

Bill, it's about time then you had a look at

  http://hebbut.net/Public.Offerings/crm114/crm114_BillY_VT_GerH.html

and maybe also the code downloadable at

  http://hebbut.net/Public.Offerings/crm114.html

as, though it may seem VT specific, this is a viable way to 'export'
the tokenizer, hasher, et al.

At least, this approach will turn up in one form or another in my
comments on your RFC.


Given such a set of just a few callbacks, libcrm114 can ditch the
complete 'encoding/language support' issue and all its hairiness as
the calling party will take care of it.
If you want to make it easy on folks, you could provide some 'often
used' setups with the core; one (cruder) way of doing it is shown by
my VT code.
A better approach is shown by, for example, the crypto layer of
OpenSSL: extra APIs which deliver preconfigured callbacks for 'common
tasks', such as (in our case) processing email using libcrm114 core
APIs.
I'm working on that.

> HOWEVER - the way the data is packed is an issue.  Right now we
> have several different packings:

NOT AN ISSUE. Take the term 'feature' and apply it to all.
Next, take the beautiful idea of the VT: you do not offer ONE feature
at a time (would be a performance hazard a la copying a file using
fgetc() ;-( ) but you always deliver 'feature bursts' from VT to
classifier.

Now take the entropy encoder: herewith we *define* one feature to be ONE BIT.

The (externalized/VT-ed) tokenizer (a.k.a. 'feature provider') always
will deliver 'feature bursts' as usual: here we'll always get 8*N
feature bursts, using N memory bytes (as 8 features fit in a byte and
we decide at design time that the smallest burst boundary is the
BYTE).

Problem solved through simply defining what the words are.

(One remaining bit: a tokenizer has to 'fit' the classifier: can't
have a 32-bit hash feature provider delivering to a 64-bit hash
feature processing classifier, now can we? ;-) --> polyvalence may
require specification of a set of 'feature providers', IFF I
understand this polyvalence thing sufficiently.)


LZ77 classifer: same trick: define 'feature' as being one RAW BYTE and
we can make it happen. That would be the simplest tokenizer ever: {
burstlen=read(handle, buf, buflen); return burstlen; }


Given your list, there currently are thus four 'feature provider'
types: 1-bit, 8-bit, 32-bit hash and 64-bit hash. All can (at least
theoretically) be piped through a VT stage; in the first two
situations you might want to use <unigram> only as that'd be a 1:1
pass-through for VT. Benefit of thinking like that: one [process]pipe
to fit 'em all --> less 'exceptions' --> lower bug risk PLUS you get
to a 80% test coverage much quicker.


> * One bit, sequence important- used by the bit entropy classifier.
>  This is identical to unaltered LATIN-1 / UTF-8 text.

Njet. Sure, what you say isn't wrong, but the fact that you address
character encodings here is (IMHO) fuzzying: bit entropy classifier
doesn't mind AT ALL what your encoding is. It can be fed RAW data
(like my sampled feeds) as well as ANY other 'data stream'. The
encoding of that stream is irrelevant on ONE CONDITION: that 'matches'
(similarities) use the same encoding/etc. as the trained samples
they're compared against. This same condition exists for all other
classifiers today as well, so this is not something specific to
BitEntropy.

Example: a trained medicine spam encoded as 7-bit (U.S.) ASCII will
NOT match the very same message *content* encoded in EBCDIC. (Anyhow,
today this is true for all classifiers in crm114; if you want
'character encoding' NOT to be [a significant] part of the
'characteristics' of the classified messages, you must ensure all
messages are converted to one single encoding before classifying them.
Today, crm114 can only do that with external (iconv(1)) help, so the
condition is no change from current crm114 practices.)


The point: character encoding is only relevant when you use regexes;
those are used today for tokenization as well as data transformations,
so 'encoding' issues pop up in both areas. libcrm114 only runs the
risk to suffer from 'tokenization' here, which is removed as an issue
by providing suitable callbacks/hooks in libcrm114 for tokenization
(or should I say: 'feature input delivery'?).


> * 32 bits, sequence unimportant - used by Hyperspace, SVM, and Neural
>  Network.  This is usually a single hash() of a feature.  Note that
>  Hyperspace sorts these into ascending order for efficiency in
>  processing.

It's hash() PLUS regex-based word-extraction: both together make up
the conversion from input data to 'feature sets', which are used as
input for all classifiers.

Remove both hash and word-extraction from the libcrm114 and we've
kicked the 'language support/Unicode/RAW/whatnot encoding' issues from
libcrm114.
(Which leaves the filenames and the pipe symbol as another 'encoding'
issue, but that can be handled the same way, using generic I/O
interfaces as la OpenSSL BIO.)


> * 64 bits, sequence unimportant - used by Markov, OSB, OSBF, and
>  Winnow.  This is actually a pair of hash() values with different
>  IVs and pipelines; the goal is to prevent birthday clashes.  We
>  MIGHT want to obsolete this format and move the classifiers
>  to 32-bit unsequenced features.

Yes. No. I dunno. 32-bit hash limits the dictionary for such
classifiers; I haven't met folks who ran into that boundary, but given
the fuss about CSS big file support ('int' on 64-bit for hash indexes
an' all) my preference leans towards offering 'proper' 64-bit hashes
as well. I'm not clear how I can adjust my VT code to stay flexible
enough while keeping the performance level, but there's sure to be a
solution for that.


> I *suspect* that if you feed a 32-bit unsorted ("as it comes from
> the VT pipe") vector into bit entropy, it would work pretty well
> too.

Yup. That makes sense to me.

Hmm... given that remark, we don't necessarily have to rigidly 'bind'
specific tokenizer/feature providers to specific 'feature' formats
(1/8/32/64-bits), but we rather should bind the tokenizer to the (is
this polyvalent???) CSS database(s): *idea:* a double indirection
coming up...: libcrm114 autodetects classifier and its setup upon
loading the CSS 'files', then through a callback/hook requests the
caller to prep the appropriate 'feature providers', each of which is
used to feed one of the polyvalent classifiers working on those CSS
files.


Or didn't I understand that you, Bill, meant with 'polyvalence' that
one could feed something like this to libcrm114:

classify (A | B) {message_to_classify]

where A could be an OSBF CSS file while B doesn't have to be OSBF but
could be a, say, BitEntropy CSS?

If the above is completely wrong, please correct me. I'm still trying
to understand what you mean with 'polyvalence' and am working based on
a set of unverified assumptions here.







-- 
Met vriendelijke groeten / Best regards,

Ger Hobbelt

--------------------------------------------------
web: http://www.hobbelt.com/
 http://www.hebbut.net/
mail: [email protected]
mobile: +31-6-11 120 978
--------------------------------------------------

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/