Fwd: general document classification using crm114

:-( me and gmail /again/. Sorry.

---------- Forwarded message ----------
From: Ger Hobbelt <[email protected]>
Date: Tue, Jul 15, 2008 at 2:36 AM
Subject: Re: [Crm114-general] general document classification using crm114
To: Hoon-Young Cho <[email protected]>


Hm, nobody piped up yet?

Okay, here goes...

On Fri, Jul 11, 2008 at 11:29 AM, Hoon-Young Cho <[email protected]> wrote:
>
> Hello, I'm a new user of crm114 and still things are quite confusing about
> crm114.
>
> I have two different kinds of documents (class A, and class B), which are
> saved in directory A and B, respectively. They are general short documents
> and not e-mails. I also have some test documents in directory C.
>
> now, I want to train a classifier with crm114, so I first made two *.css
> files
> and used mailtrainer.crm to train the *.css files and tried to test the
> classifier with
> mailclassify.crm..
> but I don't think this is a correct way of building/test a general document
> classifier.
>
> Can anybody suggest me a good & easy approach ? I'm also curious about
> applying 2-byte characters like "EUC-KR" to this problem...

First item first: your initial approach sounds reasonable to me -
especially if the 'easy approach' is what you crave.

As the well-tested scripts available with CRM114 are tailored for
email filtering, you might want to make your documents look like email
themselves, e.g. by adding headers to them. To ensure the eheaders
don't play a significant role in the classification process, give them
all the same header. (If you put the filename in the subject line, for
instance, that filename will now be part of the classification: that
might not be what you want, hence the 'all the /exactly/ same headers'
idea here.)

The other way around is to take those mail*.crm scripts and hack them
to make them 'forget' it's email they're looking at.

And I'd rather use mailreaver instead of mailclassify as Bill's been
using that one much more - if my assumptions are correct there.

Last bit of blurb on item 1: make sure the documents are 'text based',
that means: no binary formats like MSWord or PDF. Trouble with TeX PDF
and a few other sources is that it'll produce image-based PDFs too, so
that would mean your content-based classification would be right down
the drain.
Of course there are ways to cope with various document formats, but
(a) do you want/need that? and (b) it's not 'simple' and never 'easy',
except under exceptional circumstances (such as having full control
over the complete documentation writing/editing process for all
documents).


And last but not least: what do you want to 'classify'? I.e.: is the
question binary (yes=ok;no=discard) or multi way (e.g. detecting the
political preference of the writer, say, democrat, republican,
libertarian, etc.)?


> applying 2-byte characters like "EUC-KR" to this problem...

Ah-h-h-hhhh <swoon> non-roman text... and with /that/ certainly comes
- I assume again, I'm good at assuming ;-) - non-word-delimited
content (as in: English is 'delimited' as it contains explicit
whitespace; IIRC Korean does not require whitespace so regular CRM114
would feel a bit... odd.


It makes for very interesting subject matter but the short end of it is this:

Bayesian classifiers work with tokens, so they /love/ words. Those
classifiers can and have been used with 'Asian' languages, but
performance is generally worse (if I have to believe the few reports I
could find on-line) as they can only work on 'per-character' instead
of 'per-word'; that means they'll classify on a 'per-syllable' basis.
Markov-chains - standard in today's Bayes boxes - alleviate that issue
a bit, but you still lose a big deal compared to equally pimped
automatons that get fed roman, i.e. stuff with mandatory whitespace
for token delimiting.

'Fix': Use production classifiers with a custom 'regex', say '/./', to
switch to the language-independent per-syllable mode, but you will
also notice the steep drop in results quality then, I'm sure.

See TREC or Citeseer for papers on this, but there's a simple cop out
in case of CRM114:

try the 'Bit Entropy' classifier as it is not 'word oriented' in any
way and seems to perform well in laboratory at least.

Cave Canem: BE is still 'under development' as far as I'm concerned;
though Bill is the Head Master on all this and I'm just a two-bit
flunky on the loose, I /can/ state with certainty that BE only has
mirrored its megatest results on 64-bit boxes for a short period of
time in the rather distant past only. <snif> To me, that's a hint
you're stepping on a rather white area of the mappe. If you've got a
Livingstone in you, by all means, go for it!

Okay, pun aside, I'd love to see your Bit Entropy results, because
that would be an incentive to do some more with/to it, right y'all?



--
Met vriendelijke groeten / Best regards,

Ger Hobbelt

--------------------------------------------------
web: http://www.hobbelt.com/
 http://www.hebbut.net/
mail: [email protected]
mobile: +31-6-11 120 978
--------------------------------------------------



-- 
Met vriendelijke groeten / Best regards,

Ger Hobbelt

--------------------------------------------------
web: http://www.hobbelt.com/
 http://www.hebbut.net/
mail: [email protected]
mobile: +31-6-11 120 978
--------------------------------------------------

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/