percentage split versus re-evaluated complete data set

Hello,

I have observed some strange behavior in the Weka Explorer.

I have one data set with 21706 instances and classify those with a percentage split of 66%.
After learning (I used RandomForest, but others have a similar behavior) the result of the evaluation of the remaining 33% (=7380 instances) gives me 644 false positives in the confusion matrix.

But when I right click on the result list and chose "re-evaluate model on current test set" and selected prior the complete data set with 21706 instances as test set, the model gives me a way better performance with only 9 false positives in the confusion matrix.
This confuses me, since I fail to understand where the 644 false positives of the subset went. They should show up in the complete dataset shouldn't they?

I managed to reproduce this behavior with the supplied iris.arff data set.
After learning with RandomForest on a 66% split (with random seed 1) I get a confusion matrix of the test set where out of 51 instances 2 Iris-virginica are misclassified as Iris-versicolor.
When I re-evaluate the model on the complete iris.arff then I get a perfect result with zero misclassifications of all 150 instances.
Now, I fail to understand where the two misclassifications of the 33% subset have gone.

Thanks for any enlightment.
Jens

P.S: This is the output of the learning, testing and re-evaluation of iris.arff:

=== Run information ===

Scheme:       weka.classifiers.trees.RandomForest -P 100 -I 100 -num-slots 1 -K 0 -M 1.0 -V 0.001 -S 1
Relation:     iris
Instances:    150
Attributes:   5
              sepallength
              sepalwidth
              petallength
              petalwidth
              class
Test mode:    split 66.0% train, remainder test

=== Classifier model (full training set) ===

RandomForest

Bagging with 100 iterations and base learner

weka.classifiers.trees.RandomTree -K 0 -M 1.0 -V 0.001 -S 1 -do-not-check-capabilities

Time taken to build model: 0.03 seconds

=== Evaluation on test split ===

Time taken to test model on test split: 0 seconds

=== Summary ===

Correctly Classified Instances          49               96.0784 %
Incorrectly Classified Instances         2                3.9216 %
Kappa statistic                          0.9408
Mean absolute error                      0.0349
Root mean squared error                  0.1432
Relative absolute error                  7.8349 %
Root relative squared error             30.2995 %
Total Number of Instances               51

=== Detailed Accuracy By Class ===

                 TP Rate  FP Rate  Precision  Recall   F-Measure  MCC      ROC Area  PRC Area  Class
                 1,000    0,000    1,000      1,000    1,000      1,000    1,000     1,000     Iris-setosa
                 1,000    0,063    0,905      1,000    0,950      0,921    0,994     0,990     Iris-versicolor
                 0,882    0,000    1,000      0,882    0,938      0,913    0,994     0,987     Iris-virginica
Weighted Avg.    0,961    0,023    0,965      0,961    0,961      0,942    0,996     0,992

=== Confusion Matrix ===

  a  b  c   <-- classified as
15  0  0 |  a = Iris-setosa
  0 19  0 |  b = Iris-versicolor
  0  2 15 |  c = Iris-virginica


=== Re-evaluation on test set ===

User supplied test set
Relation:     iris
Instances:     unknown (yet). Reading incrementally
Attributes:   5

=== Summary ===

Correctly Classified Instances         150              100      %
Incorrectly Classified Instances         0                0      %
Kappa statistic                          1
Mean absolute error                      0.0156
Root mean squared error                  0.0628
Total Number of Instances              150

=== Detailed Accuracy By Class ===

                 TP Rate  FP Rate  Precision  Recall   F-Measure  MCC      ROC Area  PRC Area  Class
                 1,000    0,000    1,000      1,000    1,000      1,000    1,000     1,000     Iris-setosa
                 1,000    0,000    1,000      1,000    1,000      1,000    1,000     1,000     Iris-versicolor
                 1,000    0,000    1,000      1,000    1,000      1,000    1,000     1,000     Iris-virginica
Weighted Avg.    1,000    0,000    1,000      1,000    1,000      1,000    1,000     1,000

=== Confusion Matrix ===

  a  b  c   <-- classified as
50  0  0 |  a = Iris-setosa
  0 50  0 |  b = Iris-versicolor
  0  0 50 |  c = Iris-virginica

_______________________________________________
Wekalist mailing list -- [email protected]
Send posts to [email protected]
To unsubscribe send an email to [email protected]
To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html