Re: Mixed 64-bit system GerH binaries / BillYscripts --> two-sided training? YES!

Ger Hobbelt escreveu:
> dang. 550 This message matches a blacklisted regular expression
> (pharma ban) (state 18). :-( Okay, find&replaced the bloody blue pill.
> Hoped the system was using CRM114 instead of brain-lobotomized
> hardcoded rules, but no such luck.
> 
> ---------- Forwarded message ----------
> From: Ger Hobbelt <[email protected]>
> Date: Sun, Aug 31, 2008 at 4:27 PM
> Subject: Re: [Crm114-general] Mixed 64-bit system GerH binaries /
> BillYscripts --> two-sided training? YES!
> To: [email protected]
> 
> 
> Oh boy!
> 
> two-sided training, Bill? As in the API def: "you have to specify
> *all* CSS files in both your 'classify' and 'learn' statements?"
> EXACTLY what I dearly want to see in CRM114. (a) it's a congruent
> design (which always scores points with me) and (b) it'll allow me /
> others to test a few really nasty ideas regarding the OSB[F]/Markov
> classifier group: because I have VERY little time for this now, here's
> the general idea:
> 
> 
> 
> Goal: idea to improve (I hope!) the OSB/Markov-based classifiers by
> reducing the number of 'noise learn entries' in the CSS files. Noise
> reducting ==> better and more 'extreme' pR values.

I've extended OSB with EDDC, or confidence Factor - the "F" in OSBF -,
with those ideas in mind. In fact the idea in EDDC is not to reduce the
number of 'noise learn entries' but to reduce their impact, or cancel
them. It might be what you want :-).

Technical description of EDDC:
======================================================================
...
The motivation for the confidence factor is to reduce the noise
introduced by features with small counts and deemphasize those with low
class separation power [...] reduce the importance of those that occur
approximately with the same frequency in all classes.
...

   Slides on EDDC:
   http://osbf-lua.luaforge.net/papers/osbf-eddc_slides.pdf

   More technical description:
   http://osbf-lua.luaforge.net/papers/osbf-eddc.pdf
======================================================================


Overview of OSBF (OSB, EDDC and TONE-HR):
======================================================================
...
Exponential differential document count (EDDC), or confidence factor, is
an intuitively and empirically derived technique for the automatic
reduction of the influence of features with low class separation power.
The idea here is to decrease the importance of features that occur
approximately equally in both ham and spam classes.
...

   http://osbf-lua.luaforge.net/papers/OSBF-Lua_VBFeb07.pdf
======================================================================

Hope it helps.

-- 
Fidelis Assis


-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/