Re: A Q. re: upcoming CRM114 library...

  Bill,

  a quick question: will the upcoming library support feeding data
  chunk-by-chunk or would it require the complete document to be mapped
  into memory?

  This is why I ask:
  imagine a deeply-nested MIME message, that contains different text body
  parts in different encodings, parts with different binary data, possibly
  MIME-encoded in different ways (b64 and q-p), possibly residing in
  different "message/rfc822" parts at different depths. While I
  recursively parse this MIME structure, at any time I have a single
  fully-decoded (and possibly charset-translated) body part with its
  associated set of headers. But, I usually have no reason to reconstruct
  the complete message from these decoded/translated parts (and it would
  even be a wrong thing to do, from classification perspective). I'd
  rather pass the freshly obtained/decoded chunk of the message to the
  classifier along with the opaque "state structure", and instantly forget
  about it, and continue traversal of the message tree. When I am over,
  I'll ask the "state structure" about the result of classification
  (/learning).

  Will such thing be possible with the new library?

  Thanks,

  Eugene

Eugene:

No, I don't think so.  And, on cogitating on this, I don't think
you'd want to, either.

If different parts of the message are encoded with differnt MIME
tags, you might well be better off feeding those different parts 
to entirely separate classifiers, that is, to classifiers that
are set up with the proper regexes, pipelines, and algorithms
to optimize classification accuracy on those blocks, then use
a higher-level combiner (such as PCA) to recombine the results.

Here's a quick snort of the API.  Note that it's now many
classes all in one data block (db != database... it's still
fast hashing inside, no SQL):

       crm114_new_cb ()   - creates a control block; this is 
       			       set up with reasonable defaults
       crm114_setflags	       
       crm114_setregex
       crm114_setpipeline              - change setup params
       crm114_setclassdefaults
       crm114_setclasssizedefaults
       
       crm114_new_db (cb, db) - uses the control block to create
       		     	      a full datablock (might be big!)

       crm114_learn_text (outdatablock, 
       			 indatablock, classnum,  -- indatablock is const!
       			  text, textlen);

       crm114_learn_features (outdatablock,
       			 indatablock, classnum,
                         features, featlen)      -- features are int32!

       crm114_classify_text (db, text, textlen,
       			    *resultblock)   -- resultblock is a big struct

       crm114_classify_features (db, features, featurelen,
       			    *resultblock)   
			    
Everything that was in the old output text is still in the 
result block; it's a struct so you don't have to use regexes
to take it apart (but we do supply a routine that turns a resultblock
into a nice textural representation similar to what we have now.

   - Bill
       			    

------------------------------------------------------------------------------
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev