Re: dstttr

On Wed, Feb 25, 2009 at 12:49 PM, Thomas Michael Hagen
<[email protected]> wrote:
> thanks, ger! that's a big, complex answer :)

You're welcome :-)

> first, there is no liaf because one cycle on the corpus will take about
> three days. i prefer to start it again manually after that. (or do you mean
> there should be a local liaf loop until the threshold is high enough?

just a local loop; may I suggest placing it here:

            #isolate (:bool2:)
            #eval (:bool2:) /:@:(:*:improvement: > 3) + (2 > 1):/
#improvement must be at least 3 pR
            {
                eval /:@::*:improvement: > 3:/ #improvement must be at
least 3 pR
                output /Improvement high enough; NO further action taken\n/
                return
            }
            #learn <osbf microgroom refute> (:*:conc-file:) [:article:]
            #output /Improvement low. Trained back OUT of ':*:class-conc:'\n/
            #return
	    }
            liaf   # <------------------------
	}

and at the same time edited it to forego that last
<refute>-as-a-way-to-undo-training there as, yes, you can do that, but
(a) I wonder why you would want to do that, given the previous code
which wanted to purposely train the input that way and (b) <refute>
does not _exactly_ 'undo' a 'learn' operation.

> i thought i was checking against the threshold with this:

you do. It's just that you also check for a minimum improvement of 3.0
pR points produced by the learn action. Which may or may not happen.
Personally, I wouldn't check for that as it rather unpredictable;
since it is a relative check, where 'did it come from'? The initial
can be, for example, pR = 0.0 or pR = 8.0, both below a threshold of
pR == 10.0.
And then pR also can easily be -20.0 as well for particular instances.
check-and-stop on +3.0 will only make one of these arbitrary examples
pass at a later time when you classify a duplicate.
See where this is going?
(Hence the move from 2005 when this was described to 2007-2009 where
mailtrainer now only checks the hard absolute line of +threshold and
no 'doing better by 3.0' or anything like that.)


>> The DS means you learn in class C *plus* you learn <refute> the same
>> message in _all_ _the_ _other_ _classes_.
>
> ok, that sublety escaped me. i'll implement that.

And make sure you keep an option in there to test SS (Single Sided,
i.e. no <refute>-ing whatsoever, ever) so you can check which method
works best for your inputs and classifier used.


>> From my POV, double sided is only good for Winnow,
>
> that sort of makes sense, but the book said dstttr was the best for osbf,
> which confuses me.

Maybe you've seen a bit of the other conversations going on here
recently. What the book says is right. For a certain area of 'right'.
The tough bottom line nobody likes to hear is that it depends on your
input -- and that you have to doublecheck / test these assertions
using your own input data. (For email users, the defaults are
generally sound.)
When applied to the public email corpuses (and several private ones as
well) the book lists the correct conclusion. It may not apply to all
email feeds, everywhere, any time, and since you are treading further
away from the oft-tested email realm, you'd better doublecheck all
them bits. ;-)

In the beginning, back in '07, I took all that was written in there as
gospel (had to, got to start somewhere); but now I take the time to
thoroughly test all 'flavours' which come to mind and sometimes the
conclusion is 'yes, that's right' and sometimes it is not - for /my/
datasets at least. For example, one dataset here gives markovian (the
oldest classifier) as a winner over osb, while another doesn't.

Moral of the story: tests various configurations with your data; and
always consider 'best config' descriptions as mere hints which may
work, or not. Test, test, test is the order of the day.



>> extracts the name of the class to train into from the classify output.
>> Are you _really_ sure you want that? Because now false classifications
>> will be trained when their pR is below 10 (and for the wrong class).
>
> no, i don't. thanks for pointing that one out.

Usually, there's an additional [semi-]manual vetting process following
a misclassification, which precedes any learning action.
Where 'misclassification' is any of these:
1- incorrectly classified as the wrong class (which implies it crosses
the threshold)
2- classified as 'unsure' due to the pR landing below threshold,
irrespective of the class listed with that

wicked bits here to watch:
- mind the bit following the 'irrespective' word in #2 there: neither
#1 or #2 can be expected to deliver the proper class to train into
when it happens.
- #1 can be discovered 'after the fact'. Much like a misclassified
spam landing in your mailbox, or worse, or good email ending in the
spam dustbin and you getting a call why the hey you didn't read what
was sent a few days ago: false negatives and false positives
respectively.
Thus the discovery that classification is 'wrong' can be severely delayed.
Which is one reason why crm114 offers separate mail classify and
train/learn scripts: those can be called at different times. In
between, there sits a human.

>> what errors did you get?
>
> i can't remember, and now that i try to reproduce it, it doesn't show up.
> probably some stupid syntax mistake i did a few weeks back.

No worries. Next time, post the booboo; I'm always interested in
failing scripts - those get added to the testset often so I have a
repository of 'live ones' which have to fail in a controlled manner.
For instance, you may have forgotten one ':' colon in that eval
statement there and you could get the wickedest failure reports (or
sometimes none at all) - at least that was the state in spring 2008.
Haven't checked the script language that rigorously again after that
frenzy. GerH builds have a much stricter parser/interpreter to catch
this kind of stuff and still the error reports are sometimes... odd.


> so, apparently there's a lot of subtle work for me to do to get this whole
> thing right.

Yup. And wait till you are at that stage where you feel confident
enough to start testing different classifiers and options
(with/without microgroom, etc.)  ;-))



-- 
Met vriendelijke groeten / Best regards,

Ger Hobbelt

--------------------------------------------------
web:    http://www.hobbelt.com/
        http://www.hebbut.net/
mail:   [email protected]
mobile: +31-6-11 120 978
--------------------------------------------------

------------------------------------------------------------------------------
Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, CA
-OSBC tackles the biggest issue in open source: Open Sourcing the Enterprise
-Strategies to boost innovation and cut costs with open source participation
-Receive a $600 discount off the registration fee with the source code: SFAD
http://p.sf.net/sfu/XcvMzF8H