Re: Crossvalidation

Michael Hall <[email protected]> Tue, 26 Mar 2024 13:17:56 -0500
Newsgroups	gmane.comp.ai.weka
Message-ID	<[email protected]>
--===============0460546964247897611==
Content-Type: multipart/alternative;
	boundary="Apple-Mail=_2BE31AC3-DF2A-4DB6-B622-3406A9AAC0C8"

--Apple-Mail=_2BE31AC3-DF2A-4DB6-B622-3406A9AAC0C8
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=utf-8



> On Mar 25, 2024, at 8:21=E2=80=AFPM, Eibe Frank =
<[email protected]> wrote:
>=20
> The setPriors() method does not affect the learning process. It only =
sets up some statistics (such as mean target value and class =
probabilities from the training set) that are used by some of the =
evaluation metrics (e.g., root relative squared error).
> =20
> The Weka server can be used to parallelise k-fold cross-validation in =
the Explorer. In the Experimenter, you can use distributed experiments =
(where you need to choose the CrossValidationSplitResultProducer in the =
advanced mode).
> =20

What I seemed to find for Weka server was=E2=80=A6
https://github.com/pentaho-labs/weka-server

Which seems to use a scorer and not cross-validation. Also what I saw =
doing was being targeted to a java written to do normal Weka =
classification/evaluation. Code like=E2=80=A6

	static Evaluation evaluate(Classifier classifier, Instances =
trainingData) throws Exception {
		Evaluation eval =3D (Evaluation)new =
ThreadedEvaluation(trainingData);
		eval.crossValidateModel(classifier, trainingData, 10, =
new java.util.Random(1));
		System.out.println("accuracy " + eval.pctCorrect());
		return eval;
	}

This again being eventually replaced with something more like =
AdaptiveEvaluation where processing would be adjusted based on machine =
cores and available memory. Cross-validation often being done with 10 =
folds seems a natural target to try this on. I seem to remember =
discussion of situations where this actually amounted to 10x10 cross =
validations being done. If it works for that and seems beneficial I =
thought it could possibly be applied to some classifiers if their =
implementations allow parallelizing. Although which if any this would be =
true of I don=E2=80=99t currently know.=20

> It may also be that the KnowledgeFlow can run k-fold cross-validation =
in parallel, but I=E2=80=99m not sure about that.
> =20

So again Knowledge flow and Experimenter wouldn=E2=80=99t be the way I =
would probably want to go at least initially.=20

Thanks.


--Apple-Mail=_2BE31AC3-DF2A-4DB6-B622-3406A9AAC0C8
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
	charset=utf-8

<html><head><meta http-equiv=3D"content-type" content=3D"text/html; =
charset=3Dutf-8"></head><body style=3D"overflow-wrap: break-word; =
-webkit-nbsp-mode: space; line-break: after-white-space;"><br =
id=3D"lineBreakAtBeginningOfMessage"><div><br><blockquote =
type=3D"cite"><div>On Mar 25, 2024, at 8:21=E2=80=AFPM, Eibe Frank =
&lt;[email protected]&gt; wrote:</div><br =
class=3D"Apple-interchange-newline"><div><meta charset=3D"UTF-8"><div =
class=3D"WordSection1" style=3D"page: WordSection1; caret-color: rgb(0, =
0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; =
font-variant-caps: normal; font-weight: 400; letter-spacing: normal; =
text-align: start; text-indent: 0px; text-transform: none; white-space: =
normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; =
text-decoration: none;"><div style=3D"margin: 0cm; font-size: 10pt; =
font-family: Calibri, sans-serif;"><span style=3D"font-size: 11pt;">The =
setPriors() method does not affect the learning process. It only sets up =
some statistics (such as mean target value and class probabilities from =
the training set) that are used by some of the evaluation metrics (e.g., =
root relative squared error).<o:p></o:p></span></div><div style=3D"margin:=
 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span =
style=3D"font-size: 11pt;"><o:p>&nbsp;</o:p></span></div><div =
style=3D"margin: 0cm; font-size: 10pt; font-family: Calibri, =
sans-serif;"><span style=3D"font-size: 11pt;">The Weka server can be =
used to parallelise k-fold cross-validation in the Explorer. In the =
Experimenter, you can use distributed experiments (where you need to =
choose the CrossValidationSplitResultProducer in the advanced =
mode).<o:p></o:p></span></div><div style=3D"margin: 0cm; font-size: =
10pt; font-family: Calibri, sans-serif;"><span style=3D"font-size: =
11pt;"><o:p>&nbsp;</o:p></span></div></div></div></blockquote><div><br></d=
iv>What I seemed to find for Weka server was=E2=80=A6</div><div><a =
href=3D"https://github.com/pentaho-labs/weka-server">https://github.com/pe=
ntaho-labs/weka-server</a></div><div><br></div><div>Which seems to use a =
scorer and not cross-validation. Also what I saw doing was being =
targeted to a java written to do normal Weka classification/evaluation. =
Code like=E2=80=A6</div><div><br></div><div><div><span =
class=3D"Apple-tab-span" style=3D"white-space:pre">	</span>static =
Evaluation evaluate(Classifier classifier, Instances trainingData) =
throws Exception {</div><div><span class=3D"Apple-tab-span" =
style=3D"white-space:pre">		</span>Evaluation eval =3D =
(Evaluation)new ThreadedEvaluation(trainingData);</div><div><span =
class=3D"Apple-tab-span" style=3D"white-space:pre">		=
</span>eval.crossValidateModel(classifier, trainingData, 10, new =
java.util.Random(1));</div><div><span class=3D"Apple-tab-span" =
style=3D"white-space:pre">		=
</span>System.out.println("accuracy " + =
eval.pctCorrect());</div><div><span class=3D"Apple-tab-span" =
style=3D"white-space:pre">		</span>return =
eval;</div><div><span class=3D"Apple-tab-span" style=3D"white-space:pre">	=
</span>}</div><div><br></div><div>This again being eventually replaced =
with something more like AdaptiveEvaluation where processing would be =
adjusted based on machine cores and available memory. Cross-validation =
often being done with 10 folds seems a natural target to try this on. I =
seem to remember discussion of situations where this actually amounted =
to 10x10 cross validations being done. If it works for that and seems =
beneficial I thought it could possibly be applied to some classifiers if =
their implementations allow parallelizing. Although which if any this =
would be true of I don=E2=80=99t currently =
know.&nbsp;</div><div><br></div><blockquote type=3D"cite"><div><div =
class=3D"WordSection1" style=3D"page: WordSection1; caret-color: rgb(0, =
0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; =
font-variant-caps: normal; font-weight: 400; letter-spacing: normal; =
text-align: start; text-indent: 0px; text-transform: none; white-space: =
normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; =
text-decoration: none;"><div style=3D"margin: 0cm; font-size: 10pt; =
font-family: Calibri, sans-serif;"><span style=3D"font-size: 11pt;">It =
may also be that the KnowledgeFlow can run k-fold cross-validation in =
parallel, but I=E2=80=99m not sure about =
that.<o:p></o:p></span></div><div style=3D"margin: 0cm; font-size: 10pt; =
font-family: Calibri, sans-serif;"><span style=3D"font-size: =
11pt;"><o:p>&nbsp;</o:p></span></div></div></div></blockquote><br></div><d=
iv>So again Knowledge flow and Experimenter wouldn=E2=80=99t be the way =
I would probably want to go at least =
initially.&nbsp;</div><div><br></div><div>Thanks.</div><br></body></html>=

--Apple-Mail=_2BE31AC3-DF2A-4DB6-B622-3406A9AAC0C8--

--===============0460546964247897611==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
Wekalist mailing list -- [email protected]
Send posts to [email protected]
To unsubscribe send an email to [email protected]
To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html

--===============0460546964247897611==--