Re: Request for Comment: Callable CRM114 Classifiers (libcrm114)

On Wed, Sep 24, 2008 at 3:04 PM, Eugene Crosser <[email protected]> wrote:
> Bill Yerazunis wrote:
>
>>    Well, here I mean in the library API. First associate a file with an
>>    opaque object, then pass this object to `learn' and `classify' calls. In
>>    your example API, the filename itself is passed to the function.
>>
>> OK - but what does that gain us?
>
> Dunno... Maybe a "visible separation line" with the backend storage? For

To name a few gains:

Bill, given that you also talk about 'embedded' use of libcrm114,
FILE*-based I/O for input and output is severely limiting: consider
'smart routers' and such, which can store their CSS in Flash while
processing [series of] network packets for input and output:

providing hooks for arbitrary I/O allows folks to use your libcrm114
in such embedded environments.


Another target audience are the 'plugin'/'language extension'
developers wishing to add libcrm114 support to other languages (I know
Eric would be pleasantly surprised to have a PythonCRM114 ;-) ); how
long would it take before the first Perl::CRM114 would appear then,
eh? Or a php5 language extension?

Okay, so you loose the wicked coolness of the crm114 script language
(unless it's included in libcrm114 and 'enabled' by the
plugin/extension developer), but you give people added flexibility of
use.

Of course, that would mean you'd have new competition too as you'd be
entering their field: libdspam et al.


Regarding full input and partial input needed for classification: the
input processing is rather sequential, so perfect for a 'buffered I/O'
approach; whether it's FILE* I/O or another (more flexible) method
doesn't matter: with a little bit of rewriting at least the production
classifiers can run quite nicely without having the need for all input
features all at once in memory: multiple calls to the Vector Tokenizer
are a-okay, where each call would process a [partial] input buffer and
produce a series of hashes. (Yes, this disregards <UNIQUE> but then
such options can be said to just require a bigger I/O 'buffer', 's
all. Plus a generic interface can easily be made to 'limit' the amount
of input per classification if you so desire.


And let's face it: when human X grabs the libcrm114 and its API, it's
chosen for its feature set/flexibility/usability mix plus I've never
seen a development newbie produce any production-ready
plugin/extension --> I'd say you can safely assume that
plugin/embedded developers can handle the few extra callbacks/function
pointers.

Heck, when omnipresent stuff like zlib offers such things since day
one, there's a zero score for the 'learning curve' argument in my
book.
And you can always provide an extra easy wrapper for FILE* I/O; again
have a look at zlib for ideas.


Note: A nice way of providing generic I/O (memory, file, network,
etc.) is shown by the OpenSSL lib (which I use a lot, hence this
reference): the BIO-based interface is maybe not as easy as a filename
string, but it certainly offers the power sought for such interfaces
(in their case: stackable filters and arbitrary I/O channels --> for
example, you can base64 enc/decode your encrypted data, which is fed
through a zlib de/compression phase as well, all in one go --
relevance for crm114? e.g. classifying [S]MIME messages by having the
classifier as a BIO stack layer!). I think I'd be able to tweak that
one into including mmap() support as well for folks terribly hot on
performance.
In the past, I've written mobile 2D barcode I/O channel code
(contrasting file I/O) using that stuff, so plugging crm114 into your
Apple iPhone for URL / SMS classification just might be an option. ;-)

Anyhow, libcrm114 is NOT for the same target audience as crm114, IMO.
Big difference between lib usage and application usage.



> It just "feels more right" to me, but you can safely disregard my
> feelings ;-)

+1



-- 
Met vriendelijke groeten / Best regards,

Ger Hobbelt

--------------------------------------------------
web: http://www.hobbelt.com/
 http://www.hebbut.net/
mail: [email protected]
mobile: +31-6-11 120 978
--------------------------------------------------

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/