Re: Fwd: general document classification using crm114

 Thank you for your detailed and helpful answer. My task is classifying only
 two classes of documents. (normal documents vs. documents including
dirty-words)
 The documents are plain texts, not binary, not very long
 (not more than 20 words, maybe).
 I want to follow your second suggestion, hacking the mail*.crm,
 but, hmm... it's going to take some time.. frankly I hoped somebody already

 had done this.. :-)

 As a simple and first implementation, I'm using naive bayesian classifier
 written in python code, and it give me about 70~80% correctness.
 Because the amount of training data is not enough and each document
 include a lot of noisy words such as meaningless symbols..
 I expect some document preprocessing and word-tokenizing will improve
 the performance a little bit.

 Bit Entropy sounds promising..but, if documents can be tokenized (or word
 -delimited), then can BE beat other approaches? anyway, I think I have to
 try your first or second approach first before I can test the BE method.

 thanks again..

2008/7/15, Ger Hobbelt <[email protected]>:
>
> :-( me and gmail /again/. Sorry.
>
> ---------- Forwarded message ----------
> From: Ger Hobbelt <[email protected]>
> Date: Tue, Jul 15, 2008 at 2:36 AM
> Subject: Re: [Crm114-general] general document classification using crm114
> To: Hoon-Young Cho <[email protected]>
>
>
> Hm, nobody piped up yet?
>
> Okay, here goes...
>
> On Fri, Jul 11, 2008 at 11:29 AM, Hoon-Young Cho <[email protected]> wrote:
> >
> > Hello, I'm a new user of crm114 and still things are quite confusing
> about
> > crm114.
> >
> > I have two different kinds of documents (class A, and class B), which are
> > saved in directory A and B, respectively. They are general short
> documents
> > and not e-mails. I also have some test documents in directory C.
> >
> > now, I want to train a classifier with crm114, so I first made two *.css
> > files
> > and used mailtrainer.crm to train the *.css files and tried to test the
> > classifier with
> > mailclassify.crm..
> > but I don't think this is a correct way of building/test a general
> document
> > classifier.
> >
> > Can anybody suggest me a good & easy approach ? I'm also curious about
> > applying 2-byte characters like "EUC-KR" to this problem...
>
> First item first: your initial approach sounds reasonable to me -
> especially if the 'easy approach' is what you crave.
>
> As the well-tested scripts available with CRM114 are tailored for
> email filtering, you might want to make your documents look like email
> themselves, e.g. by adding headers to them. To ensure the eheaders
> don't play a significant role in the classification process, give them
> all the same header. (If you put the filename in the subject line, for
> instance, that filename will now be part of the classification: that
> might not be what you want, hence the 'all the /exactly/ same headers'
> idea here.)
>
> The other way around is to take those mail*.crm scripts and hack them
> to make them 'forget' it's email they're looking at.
>
> And I'd rather use mailreaver instead of mailclassify as Bill's been
> using that one much more - if my assumptions are correct there.
>
> Last bit of blurb on item 1: make sure the documents are 'text based',
> that means: no binary formats like MSWord or PDF. Trouble with TeX PDF
> and a few other sources is that it'll produce image-based PDFs too, so
> that would mean your content-based classification would be right down
> the drain.
> Of course there are ways to cope with various document formats, but
> (a) do you want/need that? and (b) it's not 'simple' and never 'easy',
> except under exceptional circumstances (such as having full control
> over the complete documentation writing/editing process for all
> documents).
>
>
> And last but not least: what do you want to 'classify'? I.e.: is the
> question binary (yes=ok;no=discard) or multi way (e.g. detecting the
> political preference of the writer, say, democrat, republican,
> libertarian, etc.)?
>
>
> > applying 2-byte characters like "EUC-KR" to this problem...
>
> Ah-h-h-hhhh <swoon> non-roman text... and with /that/ certainly comes
> - I assume again, I'm good at assuming ;-) - non-word-delimited
> content (as in: English is 'delimited' as it contains explicit
> whitespace; IIRC Korean does not require whitespace so regular CRM114
> would feel a bit... odd.
>
>
> It makes for very interesting subject matter but the short end of it is
> this:
>
> Bayesian classifiers work with tokens, so they /love/ words. Those
> classifiers can and have been used with 'Asian' languages, but
> performance is generally worse (if I have to believe the few reports I
> could find on-line) as they can only work on 'per-character' instead
> of 'per-word'; that means they'll classify on a 'per-syllable' basis.
> Markov-chains - standard in today's Bayes boxes - alleviate that issue
> a bit, but you still lose a big deal compared to equally pimped
> automatons that get fed roman, i.e. stuff with mandatory whitespace
> for token delimiting.
>
> 'Fix': Use production classifiers with a custom 'regex', say '/./', to
> switch to the language-independent per-syllable mode, but you will
> also notice the steep drop in results quality then, I'm sure.
>
> See TREC or Citeseer for papers on this, but there's a simple cop out
> in case of CRM114:
>
> try the 'Bit Entropy' classifier as it is not 'word oriented' in any
> way and seems to perform well in laboratory at least.
>
> Cave Canem: BE is still 'under development' as far as I'm concerned;
> though Bill is the Head Master on all this and I'm just a two-bit
> flunky on the loose, I /can/ state with certainty that BE only has
> mirrored its megatest results on 64-bit boxes for a short period of
> time in the rather distant past only. <snif> To me, that's a hint
> you're stepping on a rather white area of the mappe. If you've got a
> Livingstone in you, by all means, go for it!
>
> Okay, pun aside, I'd love to see your Bit Entropy results, because
> that would be an incentive to do some more with/to it, right y'all?
>
>
>
> --
> Met vriendelijke groeten / Best regards,
>
> Ger Hobbelt
>
> --------------------------------------------------
> web: http://www.hobbelt.com/
> http://www.hebbut.net/
> mail: [email protected]
> mobile: +31-6-11 120 978
> --------------------------------------------------
>
>
>
> --
> Met vriendelijke groeten / Best regards,
>
> Ger Hobbelt
>
> --------------------------------------------------
> web: http://www.hobbelt.com/
> http://www.hebbut.net/
> mail: [email protected]
> mobile: +31-6-11 120 978
> --------------------------------------------------
>
> -------------------------------------------------------------------------
> This SF.Net email is sponsored by the Moblin Your Move Developer's
> challenge
> Build the coolest Linux based applications with Moblin SDK & win great
> prizes
> Grand prize is a trip for two to an Open Source event anywhere in the world
> http://moblin-contest.org/redirect.php?banner_id=100&url=/
> _______________________________________________
> Crm114-general mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/crm114-general
>

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/

_______________________________________________
Crm114-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/crm114-general