Re: percentage split versus re-evaluated complete data set

> I have one data set with 21706 instances and classify those with a percentage split of 66%.
>
> After learning (I used RandomForest, but others have a similar behavior) the result of the evaluation of the remaining 33% (=7380 instances) gives me 644 false positives in the confusion matrix.
>
>
>
> But when I right click on the result list and chose “re-evaluate model on current test set” and selected prior the complete data set with 21706 instances as test set, the model gives me a way better performance with only 9 false positives in the confusion matrix.
>
> This confuses me, since I fail to understand where the 644 false positives of the subset went. They should show up in the complete dataset shouldn’t they?
>
>
>
> I managed to reproduce this behavior with the supplied iris.arff data set.
>
> After learning with RandomForest on a 66% split (with random seed 1) I get a confusion matrix of the test set where out of 51 instances 2 Iris-virginica are misclassified as Iris-versicolor.
>
> When I re-evaluate the model on the complete iris.arff then I get a perfect result with zero misclassifications of all 150 instances.
>
> Now, I fail to understand where the two misclassifications of the 33% subset have gone.

Regardless of what evaluation method you use, Weka will build (and
keep) a final model on the full dataset (unless you've turned this off
under "More options") which you can then save.
If you've performed only a "percentage split", then the "Re-evaluate
model on current test set" option will be grayed out.
If you then select the original dataset as the test set, then you will
basically evaluate a model that was trained on the full dataset on the
training set.
This is essentially the same as choosing "Use training set" as the
evaluation method.

Cheers, Peter
-- 
Peter Reutemann
Dept. of Computer Science
University of Waikato, NZ
+64 (7) 577-5304
http://www.cms.waikato.ac.nz/~fracpete/
http://www.data-mining.co.nz/
_______________________________________________
Wekalist mailing list -- [email protected]
Send posts to [email protected]
To unsubscribe send an email to [email protected]
To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html