Re: topic classifier, take 3

thanks!

the corpus is certainly not balanced. there is a lot of sports (the
corpus contains every article from ten different newspapers for the
last ten years, and as we all know; everything in there is sports or
celebrities).

however, you wouldn't expect that much tech stuff. that doesn't
necessarily mean anything, though. the classifier could have decided
on any one category to converge on, i guess.

my boss is hiring two dead chickens soon, to give the program
feedback, so i'm hoping it'll work before then *shiver* :)



On Wed, Mar 11, 2009 at 5:17 PM, Ger Hobbelt <[email protected]> wrote:
> Lookin' good, lookin' good.
>
> One tidbit:
>
>>            liaf
>>            output /Improvement never increased enough\n/
>>        } #exiting the loop because pR never improved enough
>
> That output /.../ statement after the liaf will never execute, as the
> liaf will immediately move the 'program counter' to the opening brace.
> I take it you want this instead:
>
>>            liaf
>>        } #exiting the loop because pR never improved enough
>>        output /Improvement never increased enough\n/
>
>
>
>> apart from this, i am seeing, even after having run try-to-learn for
>> quite a while, that try-to-classify 'succeeds' very confidently in
>> classifying everything as technology, even though i can clearly see
>> that it belongs to sports.
>
> Hm.... okay, wild guess #1: most of your articles are sports or tech?
> (Which means I implicitly assume:
> a) your test set is not 'balanced', i.e. the number of articles that
> should go into each category is quite different, where
> b) the majority of articles is sports and tech, implying the
> undercurrent of my remark that I expect this 'off to the wrong
> category' behaviour you see to happen to more than just these two.)
>
> Anyway, this is quite some progress and we're getting close to the
> bloody hard part of statistical document classification: what to do
> when we don't like what we get. Be prepared to wave some dead poultry.
> ;-))
>
>
> --
> Met vriendelijke groeten / Best regards,
>
> Ger Hobbelt
>
> --------------------------------------------------
> web:    http://www.hobbelt.com/
>        http://www.hebbut.net/
> mail:   [email protected]
> mobile: +31-6-11 120 978
> --------------------------------------------------
>

------------------------------------------------------------------------------
Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are
powering Web 2.0 with engaging, cross-platform capabilities. Quickly and
easily build your RIAs with Flex Builder, the Eclipse(TM)based development
software that enables intelligent coding and step-through debugging.
Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com