Re: Request for Comment: Callable CRM114 Classifiers (libcrm114)

   From: Eugene Crosser <[email protected]>

   Bill Yerazunis wrote:

   >   * If we really want thread safety (and we would need it for some
   >     applications, like running in a browser that does threads)

   =2E.. or in a milter, which has to be thread-safe by its design

   >   * File name parsing:   We need to provide a call to break the filenam=
   es
   >     string into an array of filenames

   Does it need to be in the library or can be left to the application that
   uses the library?

Well, the problem is that we we currently have this wart that
filename parsing is done in-line in every classifier - but not
_identically_.  Only mostly.  The code itself is cargo-culted and
so upgrades are haphazard at best.

The fix is to have the filename parsing at least be centralized in
one routine (which isn't even particularly hard to write).  That will
fix a number of sins...


   > - ----- PROPOSED API -----

   Maybe that is going to be "over-engineering", but I's suggest two things:=


   1. abstract classifier object from the underlying css filename.

It is already.  The classifier context that a user sees is a context
that specifies N statistics files, and some flags to be used.  The 
classifier context is "portable" and can be used to learn or
classify or reconfigure.  It is also opaque; users aren't supposed
to look inside it.

   2. allow feeding data in chunks, like is done in e.g. openssl's crypto AP=

Won't work for training-cycle or stateful classifiers.  In particular,
Hyperspace, SVM, the Neural Network, and the Bit Entropy system *need* to see 
the entire text at once; the compression classifier might be coerceable
but it's hard (it still needs "end of text" markers).

Is there a big gain associated with "data in chunks"?

   I.

   example pseudocode attached

   Eugene

   --------------000104020001080305040603
   Content-Type: text/plain;
    name="crm114.api.txt"
   Content-Transfer-Encoding: quoted-printable
   Content-Disposition: inline;
    filename="crm114.api.txt"

   typedef ... crm114_classifier;
   // to abstract classifier data from the physical file name. CSS might
   // be e.g. an area of NOR FLASH memory or some such
   typedef ... crm114_ctx;		// global context
   typedef ... crm114_learn_ctx;	// learning context (current document)
   typedef ... crm114_classify_ctx;// classifying context
   typedef ... crm114_result;	// struct for probabilities, pR etc.

   crm114_ctx global_context =3D crm114_ctx_new(void *foo);

   crm114_classifier my_classifier =3D crm114_classifier_new(char *filename)=
   ;

   crm114_learn_ctx my_doc1 =3D crm114_learn_ctx_new(
				   crm114_ctx global_context,
				   crm114_classifier my_classifier,
				   int classnumber);

   repeat
	   crm114_learn_next(crm114_learn_ctx my_doc1,
				   char *text_chunk,
				   size_t chunk_length);
   until data is over;

   crm114_learn_complete(crm114_learn_ctx my_doc1,
				   crm114_result &my_result);
   // not sure if learning should return result?

   crm114_learn_ctx_free(my_doc1); // may be combined with ..._complete


   crm114_classify_ctx my_doc2 =3D crm114_classify_ctx_new(
				   crm114_ctx global_context,
				   crm114_classifier my_classifier)

   repeat
	   crm114_classify_next(crm114_classify_ctx my_doc2,
				   char *text_chunk,
				   size_t chunk_length);
   until data is over;

   crm114_classify_complete(crm114_classify_ctx my_doc2,
				   crm114_result &my_result);
   // my_result contains result of classification

   crm114_classify_ctx_free(my_doc2); // may be combined with ..._complete

   crm114_classifier_free(my_classifier); // closes the file etc.

   crm114_ctx_free(global_context);


   --------------000104020001080305040603--

   --------------enig883D15064A21ECF3E7A12787
   Content-Type: application/pgp-signature; name="signature.asc"
   Content-Description: OpenPGP digital signature
   Content-Disposition: attachment; filename="signature.asc"

   -----BEGIN PGP SIGNATURE-----
   Version: GnuPG v1.4.6 (GNU/Linux)
   Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

   iD8DBQFI2Pipfrw/cIw6UWkRAsBGAJoDmDjt5k0128LIvNXy1mzgqRIn2gCfc9tT
   HI3QW1oVydnuD2zGr361Eek=
   =d5Qu
   -----END PGP SIGNATURE-----

   --------------enig883D15064A21ECF3E7A12787--


   --===============5574148181725773470==
   Content-Type: text/plain; charset="us-ascii"
   MIME-Version: 1.0
   Content-Transfer-Encoding: 7bit
   Content-Disposition: inline

   -------------------------------------------------------------------------
   This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
   Build the coolest Linux based applications with Moblin SDK & win great prizes
   Grand prize is a trip for two to an Open Source event anywhere in the world
   http://moblin-contest.org/redirect.php?banner_id=100&url=/
   --===============5574148181725773470==
   Content-Type: text/plain; charset="us-ascii"
   MIME-Version: 1.0
   Content-Transfer-Encoding: 7bit
   Content-Disposition: inline

   _______________________________________________
   Crm114-general mailing list
   [email protected]
   https://lists.sourceforge.net/lists/listinfo/crm114-general

   --===============5574148181725773470==--


-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/