Re: Crossvalidation
Michael Hall <[email protected]> Mon, 25 Mar 2024 20:57:59 -0500
| Newsgroups | gmane.comp.ai.weka |
|---|---|
| Message-ID | <[email protected]> |
--===============4778815361534124869== Content-Type: multipart/alternative; boundary="Apple-Mail=_1FEAE1F5-1696-4D10-BA57-C2605CF59114" --Apple-Mail=_1FEAE1F5-1696-4D10-BA57-C2605CF59114 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 > On Mar 25, 2024, at 8:21=E2=80=AFPM, Eibe Frank = <[email protected]> wrote: >=20 > The setPriors() method does not affect the learning process. It only = sets up some statistics (such as mean target value and class = probabilities from the training set) that are used by some of the = evaluation metrics (e.g., root relative squared error). > =20 > The Weka server can be used to parallelise k-fold cross-validation in = the Explorer. In the Experimenter, you can use distributed experiments = (where you need to choose the CrossValidationSplitResultProducer in the = advanced mode). > =20 > It may also be that the KnowledgeFlow can run k-fold cross-validation = in parallel, but I=E2=80=99m not sure about that. > =20 > Cheers, > Eibe > =20 It seems it might be possible to omit setPriors if it it isn=E2=80=99t = always needed but it might not be that high overhead. Maybe it can=E2=80=99= t be skipped if you don=E2=80=99t know what metrics might be used at = that point? What I was thinking about was code that looked at the number = of available processors and did parallel processing based on that. = Possibly also accounting for data size. I think the instances all need = to be in memory? So it would sort of adapt to available current machine = ability and state and try to optimize performance off of that. So probably using an Evaluation subclass. Starting out to just see if I = can just get a threaded subclass working. I=E2=80=99m not that familiar = with Weka server. I=E2=80=99ll see what I can find. Mike --Apple-Mail=_1FEAE1F5-1696-4D10-BA57-C2605CF59114 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 <html><head><meta http-equiv=3D"content-type" content=3D"text/html; = charset=3Dutf-8"></head><body style=3D"overflow-wrap: break-word; = -webkit-nbsp-mode: space; line-break: after-white-space;"><br = id=3D"lineBreakAtBeginningOfMessage"><div><br><blockquote = type=3D"cite"><div>On Mar 25, 2024, at 8:21=E2=80=AFPM, Eibe Frank = <[email protected]> wrote:</div><br = class=3D"Apple-interchange-newline"><div><meta charset=3D"UTF-8"><div = class=3D"WordSection1" style=3D"page: WordSection1; caret-color: rgb(0, = 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: 400; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;"><div style=3D"margin: 0cm; font-size: 10pt; = font-family: Calibri, sans-serif;"><span style=3D"font-size: 11pt;">The = setPriors() method does not affect the learning process. It only sets up = some statistics (such as mean target value and class probabilities from = the training set) that are used by some of the evaluation metrics (e.g., = root relative squared error).<o:p></o:p></span></div><div style=3D"margin:= 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span = style=3D"font-size: 11pt;"><o:p> </o:p></span></div><div = style=3D"margin: 0cm; font-size: 10pt; font-family: Calibri, = sans-serif;"><span style=3D"font-size: 11pt;">The Weka server can be = used to parallelise k-fold cross-validation in the Explorer. In the = Experimenter, you can use distributed experiments (where you need to = choose the CrossValidationSplitResultProducer in the advanced = mode).<o:p></o:p></span></div><div style=3D"margin: 0cm; font-size: = 10pt; font-family: Calibri, sans-serif;"><span style=3D"font-size: = 11pt;"><o:p> </o:p></span></div><div style=3D"margin: 0cm; = font-size: 10pt; font-family: Calibri, sans-serif;"><span = style=3D"font-size: 11pt;">It may also be that the KnowledgeFlow can run = k-fold cross-validation in parallel, but I=E2=80=99m not sure about = that.<o:p></o:p></span></div><div style=3D"margin: 0cm; font-size: 10pt; = font-family: Calibri, sans-serif;"><span style=3D"font-size: = 11pt;"><o:p> </o:p></span></div><div style=3D"margin: 0cm; = font-size: 10pt; font-family: Calibri, sans-serif;"><span = style=3D"font-size: 11pt;">Cheers,<o:p></o:p></span></div><div = style=3D"margin: 0cm; font-size: 10pt; font-family: Calibri, = sans-serif;"><span style=3D"font-size: = 11pt;">Eibe<o:p></o:p></span></div><div style=3D"margin: 0cm; font-size: = 10pt; font-family: Calibri, sans-serif;"><span style=3D"font-size: = 11pt;"><o:p> </o:p></span></div></div></div></blockquote><br></div><d= iv>It seems it might be possible to omit setPriors if it it isn=E2=80=99t = always needed but it might not be that high overhead. Maybe it can=E2=80=99= t be skipped if you don=E2=80=99t know what metrics might be used at = that point? What I was thinking about was code that looked at the number = of available processors and did parallel processing based on that. = Possibly also accounting for data size. I think the instances all need = to be in memory? So it would sort of adapt to available current machine = ability and state and try to optimize performance off of = that.</div><div><br></div><div>So probably using an Evaluation subclass. = Starting out to just see if I can just get a threaded subclass working. = I=E2=80=99m not that familiar with Weka server. I=E2=80=99ll see what I = can find.</div><div><br></div><div>Mike</div><br></body></html>= --Apple-Mail=_1FEAE1F5-1696-4D10-BA57-C2605CF59114-- --===============4778815361534124869== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Wekalist mailing list -- [email protected] Send posts to [email protected] To unsubscribe send an email to [email protected] To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html --===============4778815361534124869==--