Re: Advice

Aha, yes.

AI::Categorizer lets you customize the tokenization behavior to be 
however you want, by subclassing the Document class and overriding the 
tokenize() method.  You could do something like this:

{
   package My::Documents;
   @ISA = qw(AI::Categorizer::Document::Text);
   sub tokenize {
     return [split ' ', $_[1]];
   }
}
my $c = new AI::Categorizer(
               document_class => 'My::Documents',
);

...

  -Ken

On Feb 5, 2005, at 11:23 AM, Jason Armstrong wrote:

> Thanks for all the good feedback, I'll certainly be following up on it.
>
> I did find one reason why I wasn't getting good matches ... when I
> looked more carefully at the perl data structure, I found that the
> 'features' hash only contained alphabetic characters. So, for example,
> in the string 'WARRIOR 14-160 14-160', only the warrior part was being
> used. Also, with 'BMW 318i' and 'BWM 525i', the numbers were being
> ignored, and with something like 'A/T', two separate features 'a' and
> 't' were there.
>
> So my further question is how to get NaiveBayes to use white space
> separated words as features ('318i', 'a/t') and not just the individual
> alphabetic characters. Is it a simple option when calling
> new AI::Categorizer?
>
> --
> Jason Armstrong
>