Re: imageFilters: Strange behaviour
Peter holunaro <[email protected]> Fri, 28 Jun 2024 00:15:48 +0100
| Newsgroups | gmane.comp.ai.weka |
|---|---|
| Message-ID | <CAGn1Z59KoyucoqjZi18E2OvPVG8LcRujsY4m3THcms7ckyRk3g@mail.gmail.com> |
--===============0488160475405235071== Content-Type: multipart/alternative; boundary="0000000000005d1d4f061be7500f" --0000000000005d1d4f061be7500f Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Thanks again, Eibe. In this case, the best practice is to have one folder that contains *all* the images + Training set ARFF file (referencing the training set images) + Test set ARFF file (referencing the test set images). I think this should be useful and straightforward. Am I right? Regards, Peter On Thu, Jun 27, 2024 at 11:53=E2=80=AFPM Eibe Frank <[email protected].= nz> wrote: > The ARFF file you use for training should only reference those images in > that directory that you want to use for training. Similarly, the ARFF fil= e > you want to use for testing should only reference those images in that > directory that you want to use for testing. > > Cheers, > Eibe > > ------------------------------ > *From:* Peter holunaro <[email protected]> > *Sent:* Friday, June 28, 2024 4:11 AM > *To:* Weka machine learning workbench list. <[email protected]> > *Subject:* [Wekalist] Re: imageFilters: Strange behaviour > > You don't often get email from [email protected]. Learn why this is > important <https://aka.ms/LearnAboutSenderIdentification> > Thank you Eibe, for the prompt reply. > > If both training and test images are together available in the same > directory (ARFF of the test set is available in a different directory and > only has neme(s) of the test image(s), the classifier sees both during th= e > training process. This is not considered cheating while learning the data= ? > > To me, in order to have a healthy learning process, the training set > images and their ARFF file should be available in a different directory > from the test set images and their ARFF file, which is different from the > working mechanism of ImageFilter and your suggestion. > > Any clarification would be greatly appreciated. > > Regards, > Peter > > On Thu, Jun 27, 2024 at 6:06=E2=80=AFAM Eibe Frank <[email protected]= .nz> > wrote: > > This should work if your training and test images are in the same > directory, the directory that is given as a parameter in the > GenericObjectEditor when configuring the filter. > > > > Of course, the training and test data should not have any overlap. > > > > Cheers, > > Eibe > > > > *From: *Peter holunaro <[email protected]> > *Date: *Thursday, 27 June 2024 at 12:26=E2=80=AFPM > *To: *Weka machine learning workbench list. <[email protected]> > *Subject: *[Wekalist] imageFilters: Strange behaviour > > Some people who received this message don't often get email from > [email protected]. Learn why this is important > <https://aka.ms/LearnAboutSenderIdentification> > > Dear all, > > > > When working with "imageFilters" I found the following strange behaviour: > > > > I divided the data into training and test sets. The test set, as usual, i= s > totally different from the training set. Then, when running the > classification process via FilteredClassifier (classifier: J48; filter: > MultiFilter--ColorLayoutFilter + Remove the first attribute), I have an > error like this: > > > > ImageFilter Exception: was unable to read file mno001.jpg! > > > > The idea here is that the desired image is not in the training set, it is > only in the test set. But, when adding this image to the training set, > apply the classification process (highlighted up), and use it as well in > the test set (i.e., the desired image existing in both datasets). The > classification process works well and the target label is predicted > normally. > > > > I'm trying to say that the performed prediction procedure is not conducte= d > fairly when using ImageFilter since all the images in the test set need t= o > be originally in the training set. Hence, the classifier will see the > images twice--on the training and the test set--resulting in an optimisti= c > result (overfitting). > > > > Any advice to avoid such an issue is highly appreciated. > > > > Regards, > > Peter > _______________________________________________ > Wekalist mailing list -- [email protected] > Send posts to [email protected] > To unsubscribe send an email to [email protected] > To subscribe, unsubscribe, etc., visit > https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz > List etiquette: > http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html > > _______________________________________________ > Wekalist mailing list -- [email protected] > Send posts to [email protected] > To unsubscribe send an email to [email protected] > To subscribe, unsubscribe, etc., visit > https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz > List etiquette: > http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html > --0000000000005d1d4f061be7500f Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr">Thanks again, Eibe.<div><br></div><div>In this case, the b= est practice is to have one folder that contains *all* the images=C2=A0+ Tr= aining set ARFF file (referencing the training set images)=C2=A0+ Test set = ARFF file (referencing the test set images). I think this should be useful = and straightforward. Am I right?</div><div><br></div><div>Regards,=C2=A0</d= iv><div>Peter</div></div><br><div class=3D"gmail_quote"><div dir=3D"ltr" cl= ass=3D"gmail_attr">On Thu, Jun 27, 2024 at 11:53=E2=80=AFPM Eibe Frank <= <a href=3D"mailto:[email protected]">[email protected]</a>>= ; wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px= 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <div> <div style=3D"font-family:inherit;font-size:inherit;background-color:transp= arent"> The ARFF file you use for training should only reference those images in th= at directory that you want to use for training. Similarly, the ARFF file yo= u want to use for testing should only reference those images in that direct= ory that you want to use for testing.</div> <div style=3D"font-family:inherit;font-size:inherit;background-color:transp= arent"> <br> </div> <div style=3D"font-family:inherit;font-size:inherit;background-color:transp= arent"> Cheers,</div> <div style=3D"font-family:inherit;font-size:inherit;background-color:transp= arent"> Eibe</div> <br> <hr style=3D"display:inline-block;width:98%"> <div id=3D"m_8580237219141666774divRplyFwdMsg" dir=3D"ltr"><font face=3D"Ca= libri, sans-serif" style=3D"font-size:11pt" color=3D"#000000"><b>From:</b> = Peter holunaro <<a href=3D"mailto:[email protected]" target=3D"_bl= ank">[email protected]</a>><br> <b>Sent:</b> Friday, June 28, 2024 4:11 AM<br> <b>To:</b> Weka machine learning workbench list. <<a href=3D"mailto:weka= [email protected]" target=3D"_blank">[email protected]</a>&= gt;<br> <b>Subject:</b> [Wekalist] Re: imageFilters: Strange behaviour</font> <div>=C2=A0</div> </div> <div> <table border=3D"0" cellspacing=3D"0" cellpadding=3D"0" width=3D"100%" alig= n=3D"left" style=3D"border:0px;display:table;width:100%;table-layout:fixed;= float:none"> <tbody style=3D"display:block"> <tr> <td valign=3D"middle" width=3D"1px" bgcolor=3D"#A6A6A6" cellpadding=3D"7px = 2px 7px 2px" style=3D"padding:7px 2px;background-color:rgb(166,166,166)"> </td> <td valign=3D"middle" width=3D"100%" bgcolor=3D"#EAEAEA" cellpadding=3D"7px= 5px 7px 15px" color=3D"#212121" style=3D"width:100%;background-color:rgb(2= 34,234,234);padding:7px 5px 7px 15px;font-family:wf_segoe-ui_normal,"S= egoe UI","Segoe WP",Tahoma,Arial,sans-serif;font-size:12px;f= ont-weight:normal;color:rgb(33,33,33);text-align:left"> <div>You don't often get email from <a href=3D"mailto:peterholunaro@gma= il.com" target=3D"_blank">[email protected]</a>. <a href=3D"https://a= ka.ms/LearnAboutSenderIdentification" target=3D"_blank"> Learn why this is important</a></div> </td> <td valign=3D"middle" align=3D"left" width=3D"75px" bgcolor=3D"#EAEAEA" cel= lpadding=3D"7px 5px 7px 5px" color=3D"#212121" style=3D"width:75px;backgrou= nd-color:rgb(234,234,234);padding:7px 5px;font-family:wf_segoe-ui_normal,&q= uot;Segoe UI","Segoe WP",Tahoma,Arial,sans-serif;font-size:1= 2px;font-weight:normal;color:rgb(33,33,33);text-align:left"> </td> </tr> </tbody> </table> <div> <div dir=3D"ltr">Thank you Eibe, for the prompt reply. <div><br> </div> <div>If both training and test images are together available in the same di= rectory (ARFF of the test set is available in a different directory and onl= y has neme(s) of the test image(s), the classifier sees both during the tra= ining process. This is not considered cheating while learning the data?=C2=A0</div> <div><br> </div> <div>To me, in order to have a healthy learning process, the training set i= mages and their ARFF file should be available in a different directory from= the test set images and their ARFF file, which is different from the worki= ng mechanism of ImageFilter and your suggestion.=C2=A0<br> </div> <div><br> </div> <div>Any clarification would be greatly appreciated.</div> <div><br> </div> <div>Regards,=C2=A0</div> <div>Peter</div> </div> <br> <div> <div dir=3D"ltr">On Thu, Jun 27, 2024 at 6:06=E2=80=AFAM Eibe Frank <<a = href=3D"mailto:[email protected]" target=3D"_blank">eibe.frank@waika= to.ac.nz</a>> wrote:<br> </div> <blockquote style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204= ,204,204);padding-left:1ex"> <div> <div lang=3D"EN-NZ"> <div> <p><span style=3D"font-size:11pt">This should work if your training and tes= t images are in the same directory, the directory that is given as a parame= ter in the GenericObjectEditor when configuring the filter. <u></u><u></u></span></p> <p><span style=3D"font-size:11pt"><u></u>=C2=A0<u></u></span></p> <p><span style=3D"font-size:11pt">Of course, the training and test data sho= uld not have any overlap.<u></u><u></u></span></p> <p><span style=3D"font-size:11pt"><u></u>=C2=A0<u></u></span></p> <p><span style=3D"font-size:11pt">Cheers,<u></u><u></u></span></p> <p><span style=3D"font-size:11pt">Eibe<u></u><u></u></span></p> <p><span style=3D"font-size:11pt"><u></u>=C2=A0<u></u></span></p> <div id=3D"m_8580237219141666774x_m_5025650672554470779mail-editor-referenc= e-message-container"> <div> <div style=3D"border-right:none;border-bottom:none;border-left:none;border-= top:1pt solid rgb(181,196,223);padding:3pt 0cm 0cm"> <p style=3D"margin-bottom:12pt"><b><span style=3D"color:black">From: </span></b><span style=3D"color:black">Peter holunaro <<a href=3D"mailto= :[email protected]" target=3D"_blank">[email protected]</a>>= <br> <b>Date: </b>Thursday, 27 June 2024 at 12:26</span><span style=3D"font-fami= ly:Arial,sans-serif;color:black">=E2=80=AF</span><span style=3D"color:black= ">PM<br> <b>To: </b>Weka machine learning workbench list. <<a href=3D"mailto:weka= [email protected]" target=3D"_blank">[email protected]</a>&= gt;<br> <b>Subject: </b>[Wekalist] imageFilters: Strange behaviour<u></u><u></u></s= pan></p> </div> <table border=3D"0" cellspacing=3D"0" cellpadding=3D"0" align=3D"left" widt= h=3D"100%" style=3D"width:100%;display:table;float:none"> <tbody> <tr> <td style=3D"background:rgb(166,166,166);padding:5.25pt 1.5pt"></td> <td width=3D"100%" style=3D"width:100%;background:rgb(234,234,234);padding:= 5.25pt 3.75pt 5.25pt 11.25pt"> <div> <p><span style=3D"font-size:9pt;font-family:"Segoe UI",sans-serif= ;color:rgb(33,33,33)">Some people who received this message don't often= get email from <a href=3D"mailto:[email protected]" target=3D"_blank">peterholunaro@= gmail.com</a>. </span><span style=3D"color:black"><a href=3D"https://aka.ms/LearnAboutSend= erIdentification" target=3D"_blank"><span style=3D"font-size:9pt;font-famil= y:"Segoe UI",sans-serif">Learn why this is important</span></a></= span><span style=3D"font-size:9pt;font-family:"Segoe UI",sans-ser= if;color:rgb(33,33,33)"><u></u><u></u></span></p> </div> </td> <td width=3D"75" style=3D"width:56.25pt;background:rgb(234,234,234);padding= :5.25pt 3.75pt"> </td> </tr> </tbody> </table> <div> <div> <p>Dear all,=C2=A0 </p> <div> <p><u></u>=C2=A0<u></u></p> </div> <div> <p>When working with "imageFilters" I found the following strange= behaviour:</p> </div> <div> <p><u></u>=C2=A0<u></u></p> </div> <div> <p>I divided the data into training and test sets. The test set, as usual, = is totally different from the training set. Then, when running the classifi= cation process via=C2=A0FilteredClassifier (classifier: J48; filter: MultiF= ilter--ColorLayoutFilter + Remove the first attribute), I have an error like this:</p> </div> <div> <p><u></u>=C2=A0<u></u></p> </div> <div> <p>=C2=A0ImageFilter Exception: was unable to read file mno001.jpg!=C2=A0</= p> </div> <div> <p><u></u>=C2=A0<u></u></p> </div> <div> <p>The idea here is that the desired image is not in the training set, it i= s only in the test set. But, when adding this image to the training set, ap= ply the classification process (highlighted up), and use it as well in the = test set (i.e., the desired image existing in both datasets). The classification process w= orks well and the target label is predicted normally.</p> </div> <div> <p><u></u>=C2=A0<u></u></p> </div> <div> <p>I'm trying to say that the performed prediction procedure is not con= ducted fairly when using ImageFilter since all the images in the test set n= eed to be originally in the training set. Hence, the classifier will see th= e images twice--on the training and the test set--resulting in an optimistic result (overfitt= ing).=C2=A0=C2=A0</p> </div> <div> <p><u></u>=C2=A0<u></u></p> </div> <div> <p>Any advice to avoid such an issue is highly appreciated.</p> </div> <div> <p><u></u>=C2=A0<u></u></p> </div> <div> <p>Regards,=C2=A0</p> </div> <div> <p>Peter</p> </div> </div> </div> </div> </div> </div> </div> _______________________________________________<br> Wekalist mailing list -- <a href=3D"mailto:[email protected]" tar= get=3D"_blank"> [email protected]</a><br> Send posts to <a href=3D"mailto:[email protected]" target=3D"_bla= nk">[email protected]</a><br> To unsubscribe send an email to <a href=3D"mailto:[email protected]= to.ac.nz" target=3D"_blank"> [email protected]</a><br> To subscribe, unsubscribe, etc., visit <a href=3D"https://list.waikato.ac.n= z/postorius/lists/wekalist.list.waikato.ac.nz" rel=3D"noreferrer" target=3D= "_blank"> https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz</a><= br> List etiquette: <a href=3D"http://www.cs.waikato.ac.nz/~ml/weka/mailinglist= _etiquette.html" rel=3D"noreferrer" target=3D"_blank"> http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html</a><br> </div> </blockquote> </div> </div> </div> </div> _______________________________________________<br> Wekalist mailing list -- <a href=3D"mailto:[email protected]" tar= get=3D"_blank">[email protected]</a><br> Send posts to <a href=3D"mailto:[email protected]" target=3D"_bla= nk">[email protected]</a><br> To unsubscribe send an email to <a href=3D"mailto:[email protected]= to.ac.nz" target=3D"_blank">[email protected]</a><br> To subscribe, unsubscribe, etc., visit <a href=3D"https://list.waikato.ac.n= z/postorius/lists/wekalist.list.waikato.ac.nz" rel=3D"noreferrer" target=3D= "_blank">https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.a= c.nz</a><br> List etiquette: <a href=3D"http://www.cs.waikato.ac.nz/~ml/weka/mailinglist= _etiquette.html" rel=3D"noreferrer" target=3D"_blank">http://www.cs.waikato= .ac.nz/~ml/weka/mailinglist_etiquette.html</a><br> </blockquote></div> --0000000000005d1d4f061be7500f-- --===============0488160475405235071== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Wekalist mailing list -- [email protected] Send posts to [email protected] To unsubscribe send an email to [email protected] To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html --===============0488160475405235071==--