article classifier finished.

i've finished my newspaper article classifier written in crm114.

i got an accuracy of 54%, versus 59% for humans.

the measurement was done on 106 articles from 9 different categories.

6 human annotators categorized all the articles manually, and their effort
was deemed correct only when all 6 agreed on a category. when i loosened the
requirement to 5 out of 6, the human accuracy went up to 81%.

the 9 categories are:

sports,
domestic,
foreign,
economy,
politics,
culture,
entertainment,
consumer and
science and technology.

all in all, not as good as i had hoped, but not disastrous either.

the results would probably have been much better, both for humans and for
the crm program, if it were possible to place an article in more than one
category.

furthermore, the program was only trained on about three hundred or fewer
(manually categorized) articles from each category. the 'science and
technology' category was trained on only 73 articles, since no more could
conveniently be found. more training would almost certainly have increased
the accuracy.

i used a binary method, where i first sorted the whole material into sports
vs. non-sports. i then sorted the remaining material into domestic vs.
non-domestic, and so on.

the order of the binary classification could probably also have been better.
politics had a lot of false positives, so that should perhaps have been the
last (default) category, rather than one of the middle ones.

if you're interested in looking at the code, you can find it at:

http://gandalf.uib.no/aviskorpus/emneklassifisering/

ps: yes, we are so original as to have a server named gandalf ;)

any comments are, of course, welcome!

------------------------------------------------------------------------------
This SF.Net email is sponsored by the Verizon Developer Community
Take advantage of Verizon's best-in-class app development support
A streamlined, 14 day to market process makes app distribution fast and easy
Join now and get one step closer to millions of Verizon customers
http://p.sf.net/sfu/verizon-dev2dev

_______________________________________________
Crm114-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/crm114-general