Re: About the optimization of the Naive Bayes classifier.

Hi Eibe,
Thank you for your reply.


MultiScheme appears to be used to compare the performance of different classifiers. I merged the training and development sets into one training set and then used 10-folds cross-validation, which still did not work particularly well on the test set.


If I want to train a classifier model based on a train set (e.g. naive bayes) and optimize the parameters of the model using a development set in several iterations, and finally use a test set to test the performance of the model.
I would like to know how to implement it using weka.


Cheers,
Liming.






 
 
 
------------------ Original ------------------
From: &nbsp;"Eibe Frank"<[email protected]&gt;;
Date: &nbsp;Sat, May 1, 2021 05:03 PM
To: &nbsp;"Weka machine learning workbench list."<[email protected]&gt;; 

Subject: &nbsp;[Wekalist] Re: About the optimization of the Naive Bayes classifier.

&nbsp;

There is no very convenient way to do this in WEKA using a single train/validation split. However, you can use MultiScheme (https://weka.sourceforge.io/doc.stable-3-8/weka/classifiers/meta/MultiScheme.html) to implement selection using k-fold cross-validation (on the training set). This will be more robust anyway and is generally preferable unless the dataset is so large that k-fold cross-validation becomes too expensive..



Cheers,
Eibe



On Fri, Apr 30, 2021 at 11:22 AM Liming Tan <[email protected]&gt; wrote:

Hello!


A paper I read recently mentioned the use of the open-source toolkit WEKA.&nbsp;


Three data sets are used in the paper: training set, development set, and test set.&nbsp;The classifier chosen is a Naive Bayes classifier.


The original paper contains this sentence:
"The parameters of the classifier (using kernel density or normal estimator) are  optimised on the development set and applied to the test set."


But I don't find the option to use the development set in WEKA's Explorer.
Does this mean that the development set is merged into the training set?


_______________________________________________
 Wekalist mailing list -- [email protected]
 Send posts to [email protected]
 To unsubscribe send an email to [email protected]
 To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
 List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html

_______________________________________________
Wekalist mailing list -- [email protected]
Send posts to [email protected]
To unsubscribe send an email to [email protected]
To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html