Re: Request for Comment: Callable CRM114 Classifiers (libcrm114)

Bill Yerazunis wrote:

>    > - ----- PROPOSED API -----
> 
>    Maybe that is going to be "over-engineering", but I's suggest two things:=
>    1. abstract classifier object from the underlying css filename.
> 
> It is already.  The classifier context that a user sees is a context

Well, here I mean in the library API. First associate a file with an
opaque object, then pass this object to `learn' and `classify' calls. In
your example API, the filename itself is passed to the function.

>    2. allow feeding data in chunks, like is done in e.g. openssl's crypto AP=
> 
> Won't work for training-cycle or stateful classifiers.  In particular,
> Hyperspace, SVM, the Neural Network, and the Bit Entropy system *need* to see 
> the entire text at once; the compression classifier might be coerceable
> but it's hard (it still needs "end of text" markers).
> 
> Is there a big gain associated with "data in chunks"?

Well, if the classifiers really need to see the whole text at once, just
forget it.

The idea behind it is that *if* the technology allows to process data as
it comes and then throw it away, per-chunk approach allows for bigger
data without blowing up memory footprint (etc.). When milter passes the
body of the message to the user provided function, it does it in chunks
(presumably the size of network packets) as the message arrives over the
wire.

(It might be even more interesting if a classifier could provide running
values for probabilities as data is passed to it. Then it would be
possible to stop in the middle of the document and say "enough, we are
sufficiently certain that this document falls in class X" and stop
processing. But this is a different story.)

Eugene

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/

_______________________________________________
Crm114-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/crm114-general

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFI2dOyfrw/cIw6UWkRAtW5AJ9Yq6REV3jPQAY2FbPWjpUja/GXOACfdWa/
+sTWVku3aXZobSa1U3bXOL0=
=EI3+
-----END PGP SIGNATURE-----