Re: Request for Comment: Callable CRM114 Classifiers (libcrm114)

   From: Eugene Crosser <[email protected]>

   Well, here I mean in the library API. First associate a file with an
   opaque object, then pass this object to `learn' and `classify' calls. In
   your example API, the filename itself is passed to the function.

OK - but what does that gain us?  I see two calls instead of one, 
and a steeper learning curve.  Is there an upside to it versus
hiding all of the dirty business inside a single call with filename
inputs?

   >    2. allow feeding data in chunks, like is done in e.g. openssl's cryp=
   to AP=3D
   >=20
   > Won't work for training-cycle or stateful classifiers.  In particular,
   > Hyperspace, SVM, the Neural Network, and the Bit Entropy system *need* =
   to see=20
   > the entire text at once; the compression classifier might be coerceable=

   > but it's hard (it still needs "end of text" markers).
   >=20
   > Is there a big gain associated with "data in chunks"?

   Well, if the classifiers really need to see the whole text at once, just
   forget it.

Many classifiers do.

   The idea behind it is that *if* the technology allows to process data as
   it comes and then throw it away, per-chunk approach allows for bigger
   data without blowing up memory footprint (etc.). When milter passes the
   body of the message to the user provided function, it does it in chunks
   (presumably the size of network packets) as the message arrives over the
   wire.

   (It might be even more interesting if a classifier could provide running
   values for probabilities as data is passed to it. Then it would be
   possible to stop in the middle of the document and say "enough, we are
   sufficiently certain that this document falls in class X" and stop
   processing. But this is a different story.)

Indeed- but the opposite is what we have actually seen.  We actually
improve classification of spam/antispam by just looking at the
first 4 Kybtes or so and completely disregarding the rest of the
text.

In short, using more text gives you less certainty.  *sigh*.  That
is highly counterintuitive but experimentally it is true.

   - Bill Yerazunis

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/