Re: Problem Space: negatives or only positives

On Thursday 22 May 2008 22:08, Trever L. Adams wrote:
> Some texts I have seen suggest to me I want hyperspace or OSBF.
> Hyperspace is faster and uses less memory which fits what I am after.
> OSBF can be (some texts suggest they trade places) more accurate than
> Hyperspace. Any suggestions?
>

It depends on your dataset - what is best for email is not necessarily best 
for web pages for example. I did some experiments with web pages once (can't 
go into detail I'm afraid). I suggest you collect a good testing set of data 
and then test all the classifiers in, with and without the various options 
like 'unique'. See how fast and accurate they are.

Also look at normalising the HTML before passing it to the classifier (e.g. 
strip out html tags if you only care about text). If you do care about HTML 
tags then you should be aware that the default tokeniser breaks on 
whitespace, which is not much good if you have a bunch of HTML tags together 
without spaces in between. Try a regex that only allows < and > at the start 
and end of each token. 

Also, try truncating the input to different lengths. Seriously: in some tests 
I found that limiting the input to 8000 bytes actually increased accuracy as 
well as speed!

And test all reasonable combinations of the above!

- Rob
(Once blamed but now only peeks at this list occasionally)

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/