Re: imageFilters: Strange behaviour
Peter holunaro <[email protected]> Fri, 28 Jun 2024 00:33:43 +0100
| Newsgroups | gmane.comp.ai.weka |
|---|---|
| Message-ID | <CAGn1Z5_Wnj7P+0+RgDQda=JnMAdvXe9BXZgQKD4kNe_vOQHnTg@mail.gmail.com> |
--===============0323903178366674395== Content-Type: multipart/alternative; boundary="000000000000664a36061be790bb" --000000000000664a36061be790bb Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Many thanks for the confirmation. Have a nice day, Eibe. Cheers, Peter On Fri, Jun 28, 2024 at 12:25=E2=80=AFAM Eibe Frank <[email protected].= nz> wrote: > Yes, that seems a good approach. > > Cheers, > Eibe > > ________________________________________ > From: Peter holunaro <[email protected]> > Sent: Friday, June 28, 2024 11:15 AM > To: Weka machine learning workbench list. > Subject: [Wekalist] Re: imageFilters: Strange behaviour > > You don't often get email from [email protected]. Learn why this is > important<https://aka.ms/LearnAboutSenderIdentification> > Thanks again, Eibe. > > In this case, the best practice is to have one folder that contains *all* > the images + Training set ARFF file (referencing the training set images)= + > Test set ARFF file (referencing the test set images). I think this should > be useful and straightforward. Am I right? > > Regards, > Peter > > On Thu, Jun 27, 2024 at 11:53=E2=80=AFPM Eibe Frank <[email protected]= c.nz > <mailto:[email protected]>> wrote: > The ARFF file you use for training should only reference those images in > that directory that you want to use for training. Similarly, the ARFF fil= e > you want to use for testing should only reference those images in that > directory that you want to use for testing. > > Cheers, > Eibe > > ________________________________ > From: Peter holunaro <[email protected]<mailto: > [email protected]>> > Sent: Friday, June 28, 2024 4:11 AM > To: Weka machine learning workbench list. <[email protected] > <mailto:[email protected]>> > Subject: [Wekalist] Re: imageFilters: Strange behaviour > > You don't often get email from [email protected]<mailto: > [email protected]>. Learn why this is important< > https://aka.ms/LearnAboutSenderIdentification> > Thank you Eibe, for the prompt reply. > > If both training and test images are together available in the same > directory (ARFF of the test set is available in a different directory and > only has neme(s) of the test image(s), the classifier sees both during th= e > training process. This is not considered cheating while learning the data= ? > > To me, in order to have a healthy learning process, the training set > images and their ARFF file should be available in a different directory > from the test set images and their ARFF file, which is different from the > working mechanism of ImageFilter and your suggestion. > > Any clarification would be greatly appreciated. > > Regards, > Peter > > On Thu, Jun 27, 2024 at 6:06=E2=80=AFAM Eibe Frank <[email protected]= .nz > <mailto:[email protected]>> wrote: > > This should work if your training and test images are in the same > directory, the directory that is given as a parameter in the > GenericObjectEditor when configuring the filter. > > > > Of course, the training and test data should not have any overlap. > > > > Cheers, > > Eibe > > > > From: Peter holunaro <[email protected]<mailto: > [email protected]>> > Date: Thursday, 27 June 2024 at 12:26=E2=80=AFPM > To: Weka machine learning workbench list. <[email protected] > <mailto:[email protected]>> > Subject: [Wekalist] imageFilters: Strange behaviour > > Some people who received this message don't often get email from > [email protected]<mailto:[email protected]>. Learn why this > is important<https://aka.ms/LearnAboutSenderIdentification> > > Dear all, > > > > When working with "imageFilters" I found the following strange behaviour: > > > > I divided the data into training and test sets. The test set, as usual, i= s > totally different from the training set. Then, when running the > classification process via FilteredClassifier (classifier: J48; filter: > MultiFilter--ColorLayoutFilter + Remove the first attribute), I have an > error like this: > > > > ImageFilter Exception: was unable to read file mno001.jpg! > > > > The idea here is that the desired image is not in the training set, it is > only in the test set. But, when adding this image to the training set, > apply the classification process (highlighted up), and use it as well in > the test set (i.e., the desired image existing in both datasets). The > classification process works well and the target label is predicted > normally. > > > > I'm trying to say that the performed prediction procedure is not conducte= d > fairly when using ImageFilter since all the images in the test set need t= o > be originally in the training set. Hence, the classifier will see the > images twice--on the training and the test set--resulting in an optimisti= c > result (overfitting). > > > > Any advice to avoid such an issue is highly appreciated. > > > > Regards, > > Peter > > _______________________________________________ > Wekalist mailing list -- [email protected]<mailto: > [email protected]> > Send posts to [email protected]<mailto: > [email protected]> > To unsubscribe send an email to [email protected]<mailto: > [email protected]> > To subscribe, unsubscribe, etc., visit > https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz > List etiquette: > http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html > _______________________________________________ > Wekalist mailing list -- [email protected]<mailto: > [email protected]> > Send posts to [email protected]<mailto: > [email protected]> > To unsubscribe send an email to [email protected]<mailto: > [email protected]> > To subscribe, unsubscribe, etc., visit > https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz > List etiquette: > http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html > _______________________________________________ > Wekalist mailing list -- [email protected] > Send posts to [email protected] > To unsubscribe send an email to [email protected] > To subscribe, unsubscribe, etc., visit > https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz > List etiquette: > http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html > --000000000000664a36061be790bb Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr">Many thanks for the confirmation. Have a nice day, Eibe.<d= iv><br></div><div>Cheers,=C2=A0</div><div>Peter</div></div><br><div class= =3D"gmail_quote"><div dir=3D"ltr" class=3D"gmail_attr">On Fri, Jun 28, 2024= at 12:25=E2=80=AFAM Eibe Frank <<a href=3D"mailto:[email protected]= .nz">[email protected]</a>> wrote:<br></div><blockquote class=3D"= gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(20= 4,204,204);padding-left:1ex">Yes, that seems a good approach.<br> <br> Cheers,<br> Eibe<br> <br> ________________________________________<br> From: Peter holunaro <<a href=3D"mailto:[email protected]" target= =3D"_blank">[email protected]</a>><br> Sent: Friday, June 28, 2024 11:15 AM<br> To: Weka machine learning workbench list.<br> Subject: [Wekalist] Re: imageFilters: Strange behaviour<br> <br> You don't often get email from <a href=3D"mailto:[email protected]= m" target=3D"_blank">[email protected]</a>. Learn why this is importa= nt<<a href=3D"https://aka.ms/LearnAboutSenderIdentification" rel=3D"nore= ferrer" target=3D"_blank">https://aka.ms/LearnAboutSenderIdentification</a>= ><br> Thanks again, Eibe.<br> <br> In this case, the best practice is to have one folder that contains *all* t= he images + Training set ARFF file (referencing the training set images) + = Test set ARFF file (referencing the test set images). I think this should b= e useful and straightforward. Am I right?<br> <br> Regards,<br> Peter<br> <br> On Thu, Jun 27, 2024 at 11:53=E2=80=AFPM Eibe Frank <<a href=3D"mailto:e= [email protected]" target=3D"_blank">[email protected]</a><= mailto:<a href=3D"mailto:[email protected]" target=3D"_blank">eibe.f= [email protected]</a>>> wrote:<br> The ARFF file you use for training should only reference those images in th= at directory that you want to use for training. Similarly, the ARFF file yo= u want to use for testing should only reference those images in that direct= ory that you want to use for testing.<br> <br> Cheers,<br> Eibe<br> <br> ________________________________<br> From: Peter holunaro <<a href=3D"mailto:[email protected]" target= =3D"_blank">[email protected]</a><mailto:<a href=3D"mailto:peterho= [email protected]" target=3D"_blank">[email protected]</a>>><br> Sent: Friday, June 28, 2024 4:11 AM<br> To: Weka machine learning workbench list. <<a href=3D"mailto:wekalist@li= st.waikato.ac.nz" target=3D"_blank">[email protected]</a><mail= to:<a href=3D"mailto:[email protected]" target=3D"_blank">wekalis= [email protected]</a>>><br> Subject: [Wekalist] Re: imageFilters: Strange behaviour<br> <br> You don't often get email from <a href=3D"mailto:[email protected]= m" target=3D"_blank">[email protected]</a><mailto:<a href=3D"mailt= o:[email protected]" target=3D"_blank">[email protected]</a>>= ;. Learn why this is important<<a href=3D"https://aka.ms/LearnAboutSende= rIdentification" rel=3D"noreferrer" target=3D"_blank">https://aka.ms/LearnA= boutSenderIdentification</a>><br> Thank you Eibe, for the prompt reply.<br> <br> If both training and test images are together available in the same directo= ry (ARFF of the test set is available in a different directory and only has= neme(s) of the test image(s), the classifier sees both during the training= process. This is not considered cheating while learning the data?<br> <br> To me, in order to have a healthy learning process, the training set images= and their ARFF file should be available in a different directory from the = test set images and their ARFF file, which is different from the working me= chanism of ImageFilter and your suggestion.<br> <br> Any clarification would be greatly appreciated.<br> <br> Regards,<br> Peter<br> <br> On Thu, Jun 27, 2024 at 6:06=E2=80=AFAM Eibe Frank <<a href=3D"mailto:ei= [email protected]" target=3D"_blank">[email protected]</a><m= ailto:<a href=3D"mailto:[email protected]" target=3D"_blank">eibe.fr= [email protected]</a>>> wrote:<br> <br> This should work if your training and test images are in the same directory= , the directory that is given as a parameter in the GenericObjectEditor whe= n configuring the filter.<br> <br> <br> <br> Of course, the training and test data should not have any overlap.<br> <br> <br> <br> Cheers,<br> <br> Eibe<br> <br> <br> <br> From: Peter holunaro <<a href=3D"mailto:[email protected]" target= =3D"_blank">[email protected]</a><mailto:<a href=3D"mailto:peterho= [email protected]" target=3D"_blank">[email protected]</a>>><br> Date: Thursday, 27 June 2024 at 12:26=E2=80=AFPM<br> To: Weka machine learning workbench list. <<a href=3D"mailto:wekalist@li= st.waikato.ac.nz" target=3D"_blank">[email protected]</a><mail= to:<a href=3D"mailto:[email protected]" target=3D"_blank">wekalis= [email protected]</a>>><br> Subject: [Wekalist] imageFilters: Strange behaviour<br> <br> Some people who received this message don't often get email from <a hre= f=3D"mailto:[email protected]" target=3D"_blank">peterholunaro@gmail.= com</a><mailto:<a href=3D"mailto:[email protected]" target=3D"_bla= nk">[email protected]</a>>. Learn why this is important<<a href= =3D"https://aka.ms/LearnAboutSenderIdentification" rel=3D"noreferrer" targe= t=3D"_blank">https://aka.ms/LearnAboutSenderIdentification</a>><br> <br> Dear all,<br> <br> <br> <br> When working with "imageFilters" I found the following strange be= haviour:<br> <br> <br> <br> I divided the data into training and test sets. The test set, as usual, is = totally different from the training set. Then, when running the classificat= ion process via FilteredClassifier (classifier: J48; filter: MultiFilter--C= olorLayoutFilter + Remove the first attribute), I have an error like this:<= br> <br> <br> <br> =C2=A0ImageFilter Exception: was unable to read file mno001.jpg!<br> <br> <br> <br> The idea here is that the desired image is not in the training set, it is o= nly in the test set. But, when adding this image to the training set, apply= the classification process (highlighted up), and use it as well in the tes= t set (i.e., the desired image existing in both datasets). The classificati= on process works well and the target label is predicted normally.<br> <br> <br> <br> I'm trying to say that the performed prediction procedure is not conduc= ted fairly when using ImageFilter since all the images in the test set need= to be originally in the training set. Hence, the classifier will see the i= mages twice--on the training and the test set--resulting in an optimistic r= esult (overfitting).<br> <br> <br> <br> Any advice to avoid such an issue is highly appreciated.<br> <br> <br> <br> Regards,<br> <br> Peter<br> <br> _______________________________________________<br> Wekalist mailing list -- <a href=3D"mailto:[email protected]" tar= get=3D"_blank">[email protected]</a><mailto:<a href=3D"mailto:= [email protected]" target=3D"_blank">[email protected]<= /a>><br> Send posts to <a href=3D"mailto:[email protected]" target=3D"_bla= nk">[email protected]</a><mailto:<a href=3D"mailto:wekalist@li= st.waikato.ac.nz" target=3D"_blank">[email protected]</a>><br> To unsubscribe send an email to <a href=3D"mailto:[email protected]= to.ac.nz" target=3D"_blank">[email protected]</a><mailto= :<a href=3D"mailto:[email protected]" target=3D"_blank">wek= [email protected]</a>><br> To subscribe, unsubscribe, etc., visit <a href=3D"https://list.waikato.ac.n= z/postorius/lists/wekalist.list.waikato.ac.nz" rel=3D"noreferrer" target=3D= "_blank">https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.a= c.nz</a><br> List etiquette: <a href=3D"http://www.cs.waikato.ac.nz/~ml/weka/mailinglist= _etiquette.html" rel=3D"noreferrer" target=3D"_blank">http://www.cs.waikato= .ac.nz/~ml/weka/mailinglist_etiquette.html</a><br> _______________________________________________<br> Wekalist mailing list -- <a href=3D"mailto:[email protected]" tar= get=3D"_blank">[email protected]</a><mailto:<a href=3D"mailto:= [email protected]" target=3D"_blank">[email protected]<= /a>><br> Send posts to <a href=3D"mailto:[email protected]" target=3D"_bla= nk">[email protected]</a><mailto:<a href=3D"mailto:wekalist@li= st.waikato.ac.nz" target=3D"_blank">[email protected]</a>><br> To unsubscribe send an email to <a href=3D"mailto:[email protected]= to.ac.nz" target=3D"_blank">[email protected]</a><mailto= :<a href=3D"mailto:[email protected]" target=3D"_blank">wek= [email protected]</a>><br> To subscribe, unsubscribe, etc., visit <a href=3D"https://list.waikato.ac.n= z/postorius/lists/wekalist.list.waikato.ac.nz" rel=3D"noreferrer" target=3D= "_blank">https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.a= c.nz</a><br> List etiquette: <a href=3D"http://www.cs.waikato.ac.nz/~ml/weka/mailinglist= _etiquette.html" rel=3D"noreferrer" target=3D"_blank">http://www.cs.waikato= .ac.nz/~ml/weka/mailinglist_etiquette.html</a><br> _______________________________________________<br> Wekalist mailing list -- <a href=3D"mailto:[email protected]" tar= get=3D"_blank">[email protected]</a><br> Send posts to <a href=3D"mailto:[email protected]" target=3D"_bla= nk">[email protected]</a><br> To unsubscribe send an email to <a href=3D"mailto:[email protected]= to.ac.nz" target=3D"_blank">[email protected]</a><br> To subscribe, unsubscribe, etc., visit <a href=3D"https://list.waikato.ac.n= z/postorius/lists/wekalist.list.waikato.ac.nz" rel=3D"noreferrer" target=3D= "_blank">https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.a= c.nz</a><br> List etiquette: <a href=3D"http://www.cs.waikato.ac.nz/~ml/weka/mailinglist= _etiquette.html" rel=3D"noreferrer" target=3D"_blank">http://www.cs.waikato= .ac.nz/~ml/weka/mailinglist_etiquette.html</a><br> </blockquote></div> --000000000000664a36061be790bb-- --===============0323903178366674395== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Wekalist mailing list -- [email protected] Send posts to [email protected] To unsubscribe send an email to [email protected] To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html --===============0323903178366674395==--