Re: article classifier finished.

   MIME-Version: 1.0
   Date: Tue, 5 Jan 2010 15:46:20 +0100
   From: "Thomas M. Hagen" <[email protected]>
   Content-Type: multipart/mixed; boundary="===============6272764463502328668=="

   --===============6272764463502328668==
   Content-Type: multipart/alternative; boundary=0016e6d99a52a9cf2f047c6be6c5

   --0016e6d99a52a9cf2f047c6be6c5
   Content-Type: text/plain; charset=ISO-8859-1

   i've finished my newspaper article classifier written in crm114.

   i got an accuracy of 54%, versus 59% for humans.

   the measurement was done on 106 articles from 9 different categories.

   6 human annotators categorized all the articles manually, and their effort
   was deemed correct only when all 6 agreed on a category. when i loosened the
   requirement to 5 out of 6, the human accuracy went up to 81%.

How did you decide what the "absolute truth" was?  Majority
rule of the humans?  Some other method?

   the 9 categories are:

   sports,
   domestic,
   foreign,
   economy,
   politics,
   culture,
   entertainment,
   consumer and
   science and technology.

   all in all, not as good as i had hoped, but not disastrous either.

   the results would probably have been much better, both for humans and for
   the crm program, if it were possible to place an article in more than one
   category.

Maybe that's the right answer.

   furthermore, the program was only trained on about three hundred or fewer
   (manually categorized) articles from each category. the 'science and
   technology' category was trained on only 73 articles, since no more could
   conveniently be found. more training would almost certainly have increased
   the accuracy.

   i used a binary method, where i first sorted the whole material into sports
   vs. non-sports. i then sorted the remaining material into domestic vs.
   non-domestic, and so on.

Hmmm... so it's

	sports v. everything else
then
        domestic v. everything else left
then
        foreign v. everthing else left,

and so on?  OK, that's a weakness.  What you want is
instead to do a dendrogram; use PCA or exhaustive 
calculation to find the two supergroups that are easiest 
to separate- i.e. "consumer +  sci/tech + entertainment + culture"
versus everything else, then separate those.  Of course,
you will need to figure out what the best super-combinations
are.

Then, you split the super-groups apart and now you only need to
distinguish between members of the supergroup, which is usually
easier.  It's also faster.


   the order of the binary classification could probably also have been better.
   politics had a lot of false positives, so that should perhaps have been the
   last (default) category, rather than one of the middle ones.

   if you're interested in looking at the code, you can find it at:

   http://gandalf.uib.no/aviskorpus/emneklassifisering/

   ps: yes, we are so original as to have a server named gandalf ;)

   any comments are, of course, welcome!

I see you're using Hyperspace Unique.  OK.  

I don't have time to do it right now, but would it be OK if
I used your text database to try some other experiments?

  - Bill Yerazunis

------------------------------------------------------------------------------
This SF.Net email is sponsored by the Verizon Developer Community
Take advantage of Verizon's best-in-class app development support
A streamlined, 14 day to market process makes app distribution fast and easy
Join now and get one step closer to millions of Verizon customers
http://p.sf.net/sfu/verizon-dev2dev