Re: Class prediction of unlabeled data using WEKA

> I am using WEKA GUI (3.8.5) for my classification task. I have built several classification models that classify each instance to 0 or 1.
>
> The training set consists of 791 instances for class 1 and 18939 instances for class 0.
> The test set consists of 993 instances for class 1 and 3939 instances for class 0.
> MCC values of training and test set are above 0.95 for all classification models.
>
> After saving the models, I have loaded them to make predictions for unlabeled data with 84656 instances.
>
> I followed the steps below:
>
> 1. prepare unlabeled test data with notepad: insert '?' in class labels
> 2. load train data in preprocess window
> 3. load a saved model in result list in classify window
> 4. load unlabeled test data on supplied test set
> 5. click 'more options' and choose 'Plaintext' for the output predictions
> 6. click 're-evaluate model on current test set'
>
> However, as the result, all models predicted all instances in unlabeled test data as class 1.
> In summary section, all instances were positioned in 'ignored class unknown instances.
> Is the model simply overfitted to the training data? or is it due to some kind of mistakes in my workflow?

Your steps are fine. Though I'd probably use a filter for replacing
the class value rather than a text editor. You could use the
MissingValuesInjection filter with the ClassOnly plugin to remove the
class values using the following (unofficial) package:
https://github.com/fracpete/missing-values-imputation-weka-package

Since you don't have ground truth (i.e., class values), Weka cannot
use these unlabeled instances for evaluation, hence they get listed as
ignored. That's absolutely fine when hijacking the evaluation process
for just making predictions.

Are the "1" values showing up in the "predicted" column or the "actual" column?

Here is an excerpt from a J48 model that was trained on some portion
of the anneal UCI dataset and then made predictions on a dataset with
no class values (the "predicted" column changes):

    inst#     actual  predicted error prediction
        1        1:?        3:3       1
        2        1:?        3:3       0.978
        3        1:?        3:3       1
        4        1:?        3:3       1
        5        1:?        3:3       1
        6        1:?        3:3       1
        7        1:?        2:2       0.981

If the "predicted" column never changes, then your model may only ever
predict that one class.
Maybe your two datasets have quite different distributions? They are
definitely imbalanced and maybe SMOTE or something similar might help.

Cheers, Peter
-- 
Peter Reutemann
Dept. of Computer Science
University of Waikato, NZ
+64 (7) 577-5304
http://www.cms.waikato.ac.nz/~fracpete/
http://www.data-mining.co.nz/
_______________________________________________
Wekalist mailing list -- [email protected]
Send posts to [email protected]
To unsubscribe send an email to [email protected]
To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html