Re: How to the corresponding instance of a prediction

Hello Eibe,

Thanks, that was exactly what I was looking for.

Regards,

Florian van Daalen

-----Original Message-----
From: Eibe Frank <[email protected]> 
Sent: vrijdag 3 mei 2024 12:11
To: 'Weka machine learning workbench list.' <[email protected]>
Subject: [Wekalist] Re: How to the corresponding instance of a prediction

Waarschuwing: Deze email komt van buiten Maastro. Pas op met links in deze email of het openen van bijlagen.


Yes, as you have observed, the k-fold CV destroys the order, so you need to keep the "stage" attribute around but not pass it to the actual classifier (so that it does not influence the predictions). The tool to do this is the FilteredClassifier. Run the k-fold CV with the FilteredClassifier, using BayesNet as the base classifier in FilteredClassifier.

The gist of this idea is used in the following bit of Groovy code that uses WEKA:

https://jdurbin.github.io/blog/predictions_from_cv/

There should be some examples in pure Java as well because this issue has been discussed before, but I can't seem to find any bits of nice example code.

Cheers,
Eibe

________________________________________
From: Florian van Daalen <[email protected]>
Sent: Friday, May 3, 2024 9:26 PM
To: 'Weka machine learning workbench list.'
Subject: [Wekalist] Re: How to the corresponding instance of a prediction

[U ontvangt vaak geen e-mail van [email protected]. Informatie over waarom dit belangrijk is op https://aka.ms/LearnAboutSenderIdentification]

Hi Eib,

I have a dataset of cancer patients.
These patients have various stages of cancer.
As part of my final evaluation of the model I want to calculate various metrics regarding the patients in each stage.
For example, I want to calculate the true positive rate for just the stage 1 patients in the dataset and compare that the true positive rate of stage 2 patients.

In order to do this I need to know the model predictions per patient.
I try to collect this information with the following bit of code, after running crossvalidation on a model.


````
List<Prediction> predictions = new ArrayList<>();

for (int i = 0; i < instances.numInstances(); i++) {
            String label  = eval.predictions.get(i).actual;
            double prob = Double.valueOf(eval.predictions().get(i).toString().split(" ")[4]);
            String stage = instances.get(i).value(instances.attribute("stage"))
            predictions.add(new Prediction(prob, label, stage));
        }

return predictions.
````

This should result in a list of predictions, which contains the probability assigned by the model, the true label belonging to this record according to the Evaluation object, and the cancer stage as listed in the dataset.

However, eval.predictions does not maintain the order the records have in instances when cross validation is used, so the result is incorrect.

Could you tell me how I can do this correctly?

Regards,

Florian

-----Original Message-----
From: Eibe Frank <[email protected]>
Sent: vrijdag 3 mei 2024 10:19
To: 'Weka machine learning workbench list.' <[email protected]>
Subject: [Wekalist] Re: How to the corresponding instance of a prediction

Waarschuwing: Deze email komt van buiten Maastro. Pas op met links in deze email of het openen van bijlagen.


Not sure whether I fully understand exactly what you want to do, but the normal process to keep the ID attribute during evaluation while not using it for predictive purposes is to wrap your classifier (e.g., BayesNet) into weka.classifiers.meta.FilteredClassifier, configuring the FilteredClassifier with the Remove filter to remove the ID attribute before it is passed to the "base classifier".

Cheers,
Eib

________________________________________
From: Florian van Daalen <[email protected]>
Sent: Friday, May 3, 2024 8:09 PM
To: 'Weka machine learning workbench list.'
Subject: [Wekalist] Re: How to the corresponding instance of a prediction

[Sommige personen die dit bericht ontvangen, ontvangen vaak geen e-mail van [email protected]. Informatie over waarom dit belangrijk is op https://aka.ms/LearnAboutSenderIdentification]

Hello Peter,

The dataset already contains IDs, although I remove those during the training phase to avoid any weird overfitting due to their unique nature.

The main issue I'm running into is that I cannot tell which record is which when using Evaluation.Predictions.get(i).

My code basicly looks as follows:

BayesNet net = new BayesNet();
Instances data = readData("class", "resources/test.csv"); net.buildClassifier(data);

Evaluation eval = new Evaluation(data);
eval.crossValidateModel(net, data, 10, new Random(1));

If I now compare eval.predictions().get(i).actual() with data.get(i).value(data.attribute("class")) the two values are unequal. Indicating that the prediction in my eval object at i does not belong to the instance in my data object at i.

This mismatch only seems to happen when using crossvalidation.

Since the prediction objects do not contain any identifiers I am struggling to link them back to the original dataset.

Is there a way to identify the predictions at this point?

Regards,

Florian

-----Original Message-----
From: Peter Reutemann <[email protected]>
Sent: vrijdag 3 mei 2024 00:48
To: Weka machine learning workbench list. <[email protected]>
Subject: [Wekalist] Re: How to the corresponding instance of a prediction

Waarschuwing: We kunnen de herkomst van deze mail niet controleren. Pas extra op met links of het openen van bijlagen. Als u deze email niet verwacht, verwijder deze dan direct!


> I am doing an experiment with cancer data where I want to evaluate the performance of a model on various stages of cancer that are present in the dataset.
>
> Consequently after crossvalidating I want to go over the predictions and group them by cancer stage so I can calculate various statistics for each cancer stage.
>
>
> However, after I run eval.crossValidateModel() I no longer know which prediction belongs to which Instance in the dataset.
>
> Since Eval.predictions() does not provide any usefull identifiers to match it back to the instances, I originally assumed the prediction would maintain the same order as the instances in the dataset, but this is not correct.
>
> How do I determine which prediction belongs to which records?

You'd want to use an extra attribute with a unique ID.

Have a look at the following wiki entry:
https://waikato.github.io/weka-wiki/instance_id/

Cheers, Peter
--
My Open Source Blog - http://open.fracpete.org _______________________________________________
Wekalist mailing list -- [email protected] Send posts to [email protected] To unsubscribe send an email to [email protected] To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
_______________________________________________
Wekalist mailing list -- [email protected] Send posts to [email protected] To unsubscribe send an email to [email protected] To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
_______________________________________________
Wekalist mailing list -- [email protected] Send posts to [email protected] To unsubscribe send an email to [email protected] To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
_______________________________________________
Wekalist mailing list -- [email protected] Send posts to [email protected] To unsubscribe send an email to [email protected] To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
_______________________________________________
Wekalist mailing list -- [email protected] Send posts to [email protected] To unsubscribe send an email to [email protected] To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
_______________________________________________
Wekalist mailing list -- [email protected]
Send posts to [email protected]
To unsubscribe send an email to [email protected]
To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html