Re: Why does `kNearestNeighbours` return more neighbours than `m_kNN` specified?

> I think that there is an issue on IBk cross validation.
>
> Let's suppose we change the previous step 6 into this one: weka.classifiers.lazy.IBk -K 17 -W 0 -X -E -A "weka.core.neighboursearch.LinearNNSearch -A \"weka.core.EuclideanDistance -D -R first-4\" -S" -output-debug-info. In Classifier output will appear:
>
> IB1 instance-based classifier
> using 5 nearest neighbour(s) for classification
>
> But this is not true. Because that k is calculated considering all instances in dataset without splitting it in two sets: training and test. WEKA is cheating like a student that sees a test before the exam. WEKA sees test set.
> But that value will be ignored when WEKA will do predictions on test set. So where do you find the “honest” k namely it calculated based only on training set? Well, you must read the output of terminal and find the last Selected k = string. This is possible because we add the option -output-debug-info to IBk. In my case, it was 4.
>
> I think that this behaviour is misleading. It can confuse the user also because there are no info, disclaimer or pop-up about it.

There are two sections in the output:
1. === Classifier model (full training set) ===
2. === Evaluation on test split ===

The first one is generated when the option "Output model" is ticked in
the "More options" dialog. This will generate a model on the full
dataset and output its string representation. This model you can then
save to a serialized model file for later use. This will happen
regardless of evaluation method, whether train/test split,
cross-validation, etc, and will always be the same for the same
dataset as it uses the full dataset.

The second section contains the result of building a model on the 80%
and evaluating it against the 20% of the test set.

Using the -output-debug-info flag, you will see two outputs of
"Selected k = X" appear, one for the model built on the full training
set and one built on the 80% training set.

So in short, there is no issue with IBk's cross-validation.

Cheers, Peter
-- 
Peter Reutemann
Dept. of Computer Science
University of Waikato, Hamilton, NZ
Mobile +64 22 190 2375
https://www.cs.waikato.ac.nz/~fracpete/
http://www.data-mining.co.nz/
_______________________________________________
Wekalist mailing list -- [email protected]
Send posts to [email protected]
To unsubscribe send an email to [email protected]
To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html