Re: How to the corresponding instance of a prediction

Not sure whether I fully understand exactly what you want to do, but the normal process to keep the ID attribute during evaluation while not using it for predictive purposes is to wrap your classifier (e.g., BayesNet) into weka.classifiers.meta.FilteredClassifier, configuring the FilteredClassifier with the Remove filter to remove the ID attribute before it is passed to the "base classifier".

Cheers,
Eib

________________________________________
From: Florian van Daalen <[email protected]>
Sent: Friday, May 3, 2024 8:09 PM
To: 'Weka machine learning workbench list.'
Subject: [Wekalist] Re: How to the corresponding instance of a prediction

[Sommige personen die dit bericht ontvangen, ontvangen vaak geen e-mail van [email protected]. Informatie over waarom dit belangrijk is op https://aka.ms/LearnAboutSenderIdentification]

Hello Peter,

The dataset already contains IDs, although I remove those during the training phase to avoid any weird overfitting due to their unique nature.

The main issue I'm running into is that I cannot tell which record is which when using Evaluation.Predictions.get(i).

My code basicly looks as follows:

BayesNet net = new BayesNet();
Instances data = readData("class", "resources/test.csv");
net.buildClassifier(data);

Evaluation eval = new Evaluation(data);
eval.crossValidateModel(net, data, 10, new Random(1));

If I now compare eval.predictions().get(i).actual() with data.get(i).value(data.attribute("class")) the two values are unequal. Indicating that the prediction in my eval object at i does not belong to the instance in my data object at i.

This mismatch only seems to happen when using crossvalidation.

Since the prediction objects do not contain any identifiers I am struggling to link them back to the original dataset.

Is there a way to identify the predictions at this point?

Regards,

Florian

-----Original Message-----
From: Peter Reutemann <[email protected]>
Sent: vrijdag 3 mei 2024 00:48
To: Weka machine learning workbench list. <[email protected]>
Subject: [Wekalist] Re: How to the corresponding instance of a prediction

Waarschuwing: We kunnen de herkomst van deze mail niet controleren. Pas extra op met links of het openen van bijlagen. Als u deze email niet verwacht, verwijder deze dan direct!


> I am doing an experiment with cancer data where I want to evaluate the performance of a model on various stages of cancer that are present in the dataset.
>
> Consequently after crossvalidating I want to go over the predictions and group them by cancer stage so I can calculate various statistics for each cancer stage.
>
>
> However, after I run eval.crossValidateModel() I no longer know which prediction belongs to which Instance in the dataset.
>
> Since Eval.predictions() does not provide any usefull identifiers to match it back to the instances, I originally assumed the prediction would maintain the same order as the instances in the dataset, but this is not correct.
>
> How do I determine which prediction belongs to which records?

You'd want to use an extra attribute with a unique ID.

Have a look at the following wiki entry:
https://waikato.github.io/weka-wiki/instance_id/

Cheers, Peter
--
My Open Source Blog - http://open.fracpete.org _______________________________________________
Wekalist mailing list -- [email protected] Send posts to [email protected] To unsubscribe send an email to [email protected] To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
_______________________________________________
Wekalist mailing list -- [email protected]
Send posts to [email protected]
To unsubscribe send an email to [email protected]
To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
_______________________________________________
Wekalist mailing list -- [email protected]
Send posts to [email protected]
To unsubscribe send an email to [email protected]
To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html