Re: Multiple categories and more

   From: James Lee <csejl-/[email protected]>

   - As I increase the number of categories, will classification
     process slow down?  

Yes.  Unless you do some funny tree-building (which you aren't likely
to be doing) the classification time for OSB is linear in the number
of classes.  

     What about accuracy?  

Well, OSB is linear, so it's like slicing a pizza into increasingly
larger numbers of slices - small errors may become more apparent.

But in reality, the only way to know is to try it.

     Also, theoretically
     speaking, is there a limit on how many categories I can classify
     into?

Without recompiling, there's a limit of 127 classifiers per CLASSIFY
statement in all of the classifiers except SVM ( SVM is limited to 2 
classes).

If you recompile, you can set MAX_CLASSIFIERS to anything you want.


   - If the number of categories becomes an issue, will be it better
     (both performance and accuracy wise) for me to classify into two
     categories at a time and pick the highest pR?  For example,
     classify the text into cat0 and other, cat1 and other ... cat5
     and other.

What would work even better is to do an NxN confusion table and divide
your classes into two groups depending on affinity.  Things that are often
confused with each other or are closely related belong in the same
group.  For example, "business-new customers" and "business-old customers"
might be very similar and should go into the same group.  

   - The pR score for HyperSpace seems to be lower than OSB and OSBF,
     is this normal?

Yes.  Normalization of pR is an ongoing process and it's not perfect.

   - Is there a such thing as optimal statistics file size for each
     classifier?  It appears that the default size for OSB is 6M, OSBF
     is 1.1M and Hyperspace varies across different categories.

The "optimality" is more to balance on-disk footprint and disk IO time
versus accuracy.  I don't know of any classifier that is actually
going to get better accuracy when the size decreases.

   - Is it possible to "over-train" the classifier?  As I train more
     and more, since the file size is limited, does it simply throw
     away the oldest data?  Or is it more complicated than that?

Yes, you can overtrain (otherwise known as overfitting) but if you
train only on error, or only on error with a thick threshold, this is
very _very_ unlikely.  

As to throwing out data- yes, it's more complicated.  What
"microgrooming" does is to look at (first) the relevance of the data -
specifically, how many times something's been seen, and how long ago.

Words or phrases that have only ever been seen once are much more
likely to be useless (or typos) than words/phrases seen a hundred
times.  Within that group of seldom-seen (or only-seen-once )
words/phrases, we throw out the oldest first, as that one's had the
longest amount of time to be seen a second time (and thus become more
relevant).

So, that's why "microgrooming" is recommended.

    - Bill Yerazunis

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/