Re: empty texts

Fastest way around this conundrum would be, IMHO, to filter and siphon
off this fringe case before passing the goods on to CRM114.

Empty documents can, theoretically, be classified as they'll still
generate a couple of 0xdeadbeef-derived hashes, but I have no inkling
how (a) the latest code copes with zero length, and (b) it's
questionalbe whether such 'no content' feature codes are valuable
enough to train.

Anyway, the quick&dirty is to code the filter as

if empty then
   class=out
else
  call crm114 classify

The alternative is to replace the empty content with a 'magic word' or
ditto word sequence, which you're sure will not occur in any other
document, e.g.

if doc.empty then
  doc = "my_very_magic_and_frowned_upon_hacky_word_ho_ho_ho"
endif
call crm114 classify doc


Anyway, you get the drift... ;-)


On Wed, Jul 22, 2009 at 9:24 AM, Thomas Michael
Hagen<[email protected]> wrote:
> is there a technique to classify empty instances (mails or articles
> without any text in them at all)?
>
> i'm having a bit of trouble getting my categorizer to recognize the
> empty newspaper articles in my corpus as 'out'. it keeps classifying
> them as 'con'(sumer), which is the first category in my list (and also
> the first alphabetically).
>
> will learning (training) these empty articles as 'out' help at all? so
> far, it doesn't seem so.
>
> ------------------------------------------------------------------------------
> _______________________________________________
> Crm114-general mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/crm114-general
>
>



-- 
Met vriendelijke groeten / Best regards,

Ger Hobbelt

--------------------------------------------------
web:    http://www.hobbelt.com/
        http://www.hebbut.net/
mail:   [email protected]
mobile: +31-6-11 120 978
--------------------------------------------------

------------------------------------------------------------------------------