Re: crm filter cleanup on productive system

On Mon, Feb 23, 2009 at 6:28 AM, Frank Doege <fdoege-zZ82ZiX1S8lUvkYWv5dGcgC/[email protected]> wrote:
> In a Mailfilter concept, wouldnt it make sense to filter first by any
> conventional method like markov etc. the more significant content parts
> an then train that on the nn ? this would only work for classifying
> mail, but anyways for other purposes like classifying text the NN has to
>  look different.

Wellllll <waggles hand> the trouble with cascading classifiers like
that (or other cascade styles) is that the front stage is just another
statistical process, which can find the proper results, but also makes
mistakes. The question then becomes: can a front stage (say:
bayesian/markovian) produce meta-data of sufficient quality that it'd
'surpass' the ability of doing the same by the follow-up stage? If so,
the front stage is the better one anyway, so there's no need for the
follow up stage as it can only reduce performance, while, when the
answer is 'maybe' or 'no', one can argue the front-stage is
'polluting' instead of aiding the follow-up stage. (That's a bit
black&white, but it's close enough to reality.)

I've been thinking about multi-stage classifiers too, but the deeper I
go into the classifier technology, the less they look like 'the 'Right
Thing', unless you have *very* particular needs.

A cascade _could_ be contemplated for reasons of run-time speed, but
the math becomes rather complex that way. Mixing, say, Bayesian and NN
in a cascade that way also will very probably result in giving you the
sum of the drawbacks and only the benefits left over after that: NN
can learn certain types of patterns, which are very hard on a Bayes
box; putting a Bayes box at front to check for the 'significant
content' propagates the Bayes-inability to learn those specific types
of input to the NN classifier, thus adding one inability to the list
of NN itself.


If I sound vague (and I probably do), that's because I'm still a
little vague about cascades myself.
One test (out of several) with cascades was successful, in a way, but
that was a cascade which consisted of a (fast) Markovian front stage
with rather high thresholds, which was supposed to pick out the 'sure
things' from the input stream, while an experimental (slower)
classifier was used to classify the 'unsures' from the first stage.
Training was a real bugger, though, as both stages can make mistakes
such as false positives and false negatives and then the interesting
question becomes: which stages will get trained now? It gets rather
wicked when you also consider the 'unsures' from both stages: should
you learn stage 1 following stage 2 verdicts? Who gets trained with
the manually classified 'unsures' that popped out the back end of
stage 2? Etc.
And I am still not 100% sure this crafty setup did any better than a
single (NN) classifier would, as it's only one test that got
acceptable results and it wasn't a really 'scientific' setup anyway;
more like multiple rounds of mess&test while checking my rig and a few
odd ideas.

Anyway, nice thoughts, but motto of the day is: better stay away until
you _really_ know your input feeds and your classifiers and why you
need another one. ;-)

>
> P.S.: after resetting and updating the crm performes so well like at the
> beginning and that after training about 100 mails, dont know why.

That's most probably due to the threshold settings in mailfilter.cf
having changed since Winter '07, plus, though rather improbable, some
code changes / fixes in mailtrainer.crm which effect the training
scheme (as those effects should only be visible when batch-training
(IIRC), which is not the default behaviour for the mailfiltering
scripts).



-- 
Met vriendelijke groeten / Best regards,

Ger Hobbelt

--------------------------------------------------
web:    http://www.hobbelt.com/
        http://www.hebbut.net/
mail:   [email protected]
mobile: +31-6-11 120 978
--------------------------------------------------

------------------------------------------------------------------------------
Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, CA
-OSBC tackles the biggest issue in open source: Open Sourcing the Enterprise
-Strategies to boost innovation and cut costs with open source participation
-Receive a $600 discount off the registration fee with the source code: SFAD
http://p.sf.net/sfu/XcvMzF8H