Crossvalidation
Michael Hall <[email protected]> Sat, 23 Mar 2024 21:06:41 -0500
| Newsgroups | gmane.comp.ai.weka |
|---|---|
| Message-ID | <[email protected]> |
--===============2118456884372084234==
Content-Type: multipart/alternative;
boundary="Apple-Mail=_753E79C1-0799-4E0A-93D5-AAB9C979168C"
--Apple-Mail=_753E79C1-0799-4E0A-93D5-AAB9C979168C
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
charset=utf-8
I have been wondering about the possibility of doing cross validation =
threaded. My current machine supposedly has lots more cores than I=E2=80=99=
ve had before.=20
One thing that would of course be required is the evaluation of each =
fold be completely independent of the others. No serial dependencies.=20
Related to this I wondered if setPriors was such a dependency, where one =
fold somehow depended on information obtained from prior runs.=20
This appears to always be done.
// Do the folds
for (int i =3D 0; i < numFolds; i++) {
Instances train =3D data.trainCV(numFolds, i, random);
setPriors(train);
I decided to check with ChatGPT. Not a Weka expert but it can do =
impressive things these days right?
> In cross-validation, prior probabilities are often used in certain =
algorithms or techniques that involve probabilistic modeling, such as =
Naive Bayes classifiers or Bayesian inference methods. These algorithms =
may require knowledge of the prior probabilities of the classes in order =
to make predictions or estimate parameters.
>=20
> However, in many cases, cross-validation is performed without =
explicitly incorporating prior probabilities. Instead, the focus is on =
evaluating the performance of a model or algorithm based on the observed =
data, without assuming any specific prior beliefs about the class =
distribution.
>=20
This would seem to suggest that it=E2=80=99s prior on the initial data =
classes and not on the prior folds. It also seems to suggest that maybe =
it wouldn=E2=80=99t be necessary for all classifiers all the time? It =
would seem that for stratified folds it at least wouldn=E2=80=99t need =
to be done more than once?
Does this seem correct? Or how am I misunderstanding?=20
=20=
--Apple-Mail=_753E79C1-0799-4E0A-93D5-AAB9C979168C
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
charset=utf-8
<html><head><meta http-equiv=3D"content-type" content=3D"text/html; =
charset=3Dutf-8"></head><body style=3D"overflow-wrap: break-word; =
-webkit-nbsp-mode: space; line-break: after-white-space;">I have been =
wondering about the possibility of doing cross validation threaded. My =
current machine supposedly has lots more cores than I=E2=80=99ve had =
before. <div><br></div><div>One thing that would of course be =
required is the evaluation of each fold be completely independent of the =
others. No serial dependencies. </div><div><br></div><div>Related =
to this I wondered if setPriors was such a dependency, where one fold =
somehow depended on information obtained from prior =
runs. </div><div><br></div><div>This appears to always be =
done.</div><div><br></div><div> // Do the =
folds</div><div> for (int i =3D 0; i < numFolds; i++) =
{</div><div> Instances train =3D =
data.trainCV(numFolds, i, random);</div><div> =
setPriors(train);</div><div><br></div><div>I decided to check with =
ChatGPT. Not a Weka expert but it can do impressive things these days =
right?</div><div><br></div><blockquote type=3D"cite"><p =
style=3D"white-space: pre-wrap; border: 0px solid rgb(227, 227, 227); =
box-sizing: border-box; margin: 1.25em 0px; caret-color: rgb(13, 13, =
13); color: rgb(13, 13, 13); font-family: S=C3=B6hne, ui-sans-serif, =
system-ui, -apple-system, "Segoe UI", Roboto, Ubuntu, =
Cantarell, "Noto Sans", sans-serif, "Helvetica =
Neue", Arial, "Apple Color Emoji", "Segoe UI =
Emoji", "Segoe UI Symbol", "Noto Color =
Emoji";">In cross-validation, prior probabilities are often used in =
certain algorithms or techniques that involve probabilistic modeling, =
such as Naive Bayes classifiers or Bayesian inference methods. These =
algorithms may require knowledge of the prior probabilities of the =
classes in order to make predictions or estimate parameters.</p><p =
style=3D"white-space: pre-wrap; border: 0px solid rgb(227, 227, 227); =
box-sizing: border-box; margin: 1.25em 0px; caret-color: rgb(13, 13, =
13); color: rgb(13, 13, 13); font-family: S=C3=B6hne, ui-sans-serif, =
system-ui, -apple-system, "Segoe UI", Roboto, Ubuntu, =
Cantarell, "Noto Sans", sans-serif, "Helvetica =
Neue", Arial, "Apple Color Emoji", "Segoe UI =
Emoji", "Segoe UI Symbol", "Noto Color =
Emoji";">However, in many cases, cross-validation is performed =
without explicitly incorporating prior probabilities. Instead, the focus =
is on evaluating the performance of a model or algorithm based on the =
observed data, without assuming any specific prior beliefs about the =
class distribution.</p></blockquote><div>This would seem to suggest that =
it=E2=80=99s prior on the initial data classes and not on the prior =
folds. It also seems to suggest that maybe it wouldn=E2=80=99t be =
necessary for all classifiers all the time? It would seem that for =
stratified folds it at least wouldn=E2=80=99t need to be done more than =
once?</div><div><br></div><div>Does this seem correct? Or how am I =
misunderstanding? </div><div><br></div><div><br></div><div> &nbs=
p;</div></body></html>=
--Apple-Mail=_753E79C1-0799-4E0A-93D5-AAB9C979168C--
--===============2118456884372084234==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
_______________________________________________
Wekalist mailing list -- [email protected]
Send posts to [email protected]
To unsubscribe send an email to [email protected]
To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
--===============2118456884372084234==--