Re: Problem of large data sets
Shu-Ju Tu <[email protected]> Fri, 19 Apr 2024 08:53:16 +0800
| Newsgroups | gmane.comp.ai.weka |
|---|---|
| Message-ID | <CABaQXBuefwi9mvZBJ4bfnD1L=K3RRbTX0-+z8sxZbky6vChAPg@mail.gmail.com> |
--===============8361112038478535325== Content-Type: multipart/alternative; boundary="00000000000006fb450616688409" --00000000000006fb450616688409 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Thank you very much for the reply to our problem. Yes and the difference is in the sample size of training data sets. #1 sample n=3D100 #2 sample n=3D200 #3 sample n=3D300 #4 sample n>900 Two quick follow-up questions: (1) A large data set will take a lot longer computing time of going through the feature selection process correct? (2) For a large data set, do I need to have any special consideration of parameter settings concerning Random Forest/AdaBoostM1 learning algorithms? Best regards, Shu-Ju Eibe Frank <[email protected]> =E6=96=BC 2024=E5=B9=B44=E6=9C=8819= =E6=97=A5 =E9=80=B1=E4=BA=94 =E4=B8=8A=E5=8D=887:05=E5=AF=AB=E9=81=93=EF=BC= =9A > I assume you use the same test data in all three cases and only the > training data changes. For IID data, this kind of behaviour should not > happen with well-behaved learning algorithms such as RandomForest and > AdaBoostM1. > > > > Cheers, > > Eibe > > > > *From: *Shu-Ju Tu <[email protected]> > *Date: *Thursday, 18 April 2024 at 8:01=E2=80=AFPM > *To: *Weka machine learning workbench list. <[email protected]> > *Subject: *[Wekalist] Re: Problem of large data sets > > Hello Thank you very much for sharing your information. > > > > Our data were obtained from the same PET imaging machine and identical > settings. > > I believe our medical physicists routinely perform QA of high quality for > this machine. > > > > Previously I was thinking that is the problem of a large number of data > set (n>800). > > So that large number of data set (> 800) actually was not an issue? > > > > Shu-Ju > > > > Ulrich Mayring <[email protected]> =E6=96=BC 2024=E5=B9=B44=E6=9C=88= 18=E6=97=A5 =E9=80=B1=E5=9B=9B =E4=B8=8A=E5=8D=888:59=E5=AF=AB=E9=81=93=EF= =BC=9A > > Am 17.04.24 um 05:00 schrieb Shu-Ju Tu: > > Hi dear Weka development team staff: > > > > I have a problem of getting low predictive accuracy when running a larg= e > > data set. > > > > Here is the story and thank you for the patient in advance: > > We started a small data set (n=3D100) last year. > > It is a 2-class supervised data set and the class is evenly distributed > > 50-50. > > The correctly predictive accuracy on training after feature selection > > and test data sets is about 85%. > > We have tried RandomForest and AdaBoostM1. > > Then we increased the data set to n=3D200 (later 300) and were getting > > about similar predictive results. > > Then recently we increased to n=3D800 and were getting very low accurac= y > > of 60%. > > > > Are there something we can do and try to improve on the results? > > Maybe your new data is significantly different from the old data. If so, > you could try to retrain your model on the new data. > > I had a situation like that where I was looking at manufacturing data. > Then they reconfigured / optimised the machine and the data changed > enough to make my model useless. > > > _______________________________________________ > Wekalist mailing list -- [email protected] > Send posts to [email protected] > To unsubscribe send an email to [email protected] > To subscribe, unsubscribe, etc., visit > https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz > List etiquette: > http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html > > _______________________________________________ > Wekalist mailing list -- [email protected] > Send posts to [email protected] > To unsubscribe send an email to [email protected] > To subscribe, unsubscribe, etc., visit > https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz > List etiquette: > http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html > --00000000000006fb450616688409 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div>Thank you very much for the reply to our problem.</di= v><div>Yes and the difference is in the sample size of training data sets.<= /div><div>#1 sample n=3D100<br></div><div>#2 sample n=3D200<br></div><div>#= 3 sample n=3D300<br></div><div>#4 sample n>900</div><div><br></div><div>= Two quick follow-up questions:</div><div><br></div><div>(1) A large data se= t will take a lot longer computing time of going through the feature select= ion process correct?</div><div>(2) For a large data set, do I need to have = any special consideration of parameter settings concerning Random Forest/Ad= aBoostM1 learning algorithms?</div><div><br></div><div>Best regards,</div><= div>Shu-Ju<br></div><div><br></div><div><br></div><div><br></div><div><br><= /div><div><br></div><div><br></div><div><br></div><div><br></div></div><br>= <div class=3D"gmail_quote"><div dir=3D"ltr" class=3D"gmail_attr">Eibe Frank= <<a href=3D"mailto:[email protected]">[email protected]</= a>> =E6=96=BC 2024=E5=B9=B44=E6=9C=8819=E6=97=A5 =E9=80=B1=E4=BA=94 =E4= =B8=8A=E5=8D=887:05=E5=AF=AB=E9=81=93=EF=BC=9A<br></div><blockquote class= =3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rg= b(204,204,204);padding-left:1ex"><div class=3D"msg-6909737469120622509"> <div lang=3D"EN-NZ" style=3D"overflow-wrap: break-word;"> <div class=3D"m_-6909737469120622509WordSection1"> <p class=3D"MsoNormal"><span style=3D"font-size:11pt">I assume you use the = same test data in all three cases and only the training data changes. For I= ID data, this kind of behaviour should not happen with well-behaved learnin= g algorithms such as RandomForest and AdaBoostM1.<u></u><u></u></span></p> <p class=3D"MsoNormal"><span style=3D"font-size:11pt"><u></u>=C2=A0<u></u><= /span></p> <p class=3D"MsoNormal"><span style=3D"font-size:11pt">Cheers,<u></u><u></u>= </span></p> <p class=3D"MsoNormal"><span style=3D"font-size:11pt">Eibe<u></u><u></u></s= pan></p> <p class=3D"MsoNormal"><span style=3D"font-size:11pt"><u></u>=C2=A0<u></u><= /span></p> <div id=3D"m_-6909737469120622509mail-editor-reference-message-container"> <div> <div style=3D"border-width:1pt medium medium;border-style:solid none none;b= order-color:rgb(181,196,223) currentcolor currentcolor;padding:3pt 0cm 0cm"= > <p class=3D"MsoNormal" style=3D"margin-bottom:12pt"><b><span style=3D"color= :black">From: </span></b><span style=3D"color:black">Shu-Ju Tu <<a href=3D"mailto:sjtu= @gap.cgu.edu.tw" target=3D"_blank">[email protected]</a>><br> <b>Date: </b>Thursday, 18 April 2024 at 8:01</span><span style=3D"font-fami= ly:"Arial",sans-serif;color:black">=E2=80=AF</span><span style=3D= "color:black">PM<br> <b>To: </b>Weka machine learning workbench list. <<a href=3D"mailto:weka= [email protected]" target=3D"_blank">[email protected]</a>&= gt;<br> <b>Subject: </b>[Wekalist] Re: Problem of large data sets<u></u><u></u></sp= an></p> </div> <div> <div> <p class=3D"MsoNormal">Hello Thank you very much for sharing your informati= on.<u></u><u></u></p> </div> <div> <p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p> </div> <div> <p class=3D"MsoNormal">Our data were obtained from the same PET imaging mac= hine and identical settings.<u></u><u></u></p> </div> <div> <p class=3D"MsoNormal">I believe our medical physicists routinely perform Q= A of high quality for this machine.<u></u><u></u></p> </div> <div> <p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p> </div> <div> <p class=3D"MsoNormal">Previously I was thinking that is the problem of a l= arge number of data set (n>800).<u></u><u></u></p> </div> <div> <p class=3D"MsoNormal">So that large number of data set (> 800) actually= was not an issue?<u></u><u></u></p> </div> <div> <p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p> </div> <div> <p class=3D"MsoNormal">Shu-Ju<u></u><u></u></p> </div> </div> <p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p> <div> <div> <p class=3D"MsoNormal">Ulrich Mayring <<a href=3D"mailto:ulrich.mayring@= isys.de" target=3D"_blank">[email protected]</a>> <span lang=3D"EN-GB" style=3D"font-family:"MS Gothic"">=E6=96=BC<= /span> 2024<span lang=3D"EN-GB" style=3D"font-family:"MS Gothic""= >=E5=B9=B4</span>4<span lang=3D"EN-GB" style=3D"font-family:"MS Gothic= "">=E6=9C=88</span>18<span lang=3D"EN-GB" style=3D"font-family:"M= S Gothic"">=E6=97=A5</span><span lang=3D"EN-GB"> </span><span lang=3D"EN-GB" style=3D"font-family:"MS Gothic"">=E9= =80=B1=E5=9B=9B</span><span lang=3D"EN-GB"> </span><span lang=3D"EN-GB" style=3D"font-family:"MS Gothic"">=E4= =B8=8A=E5=8D=88</span>8:59<span lang=3D"EN-GB" style=3D"font-family:"M= S Gothic"">=E5=AF=AB=E9=81=93=EF=BC=9A</span><u></u><u></u></p> </div> <blockquote style=3D"border-width:medium medium medium 1pt;border-style:non= e none none solid;border-color:currentcolor currentcolor currentcolor rgb(2= 04,204,204);padding:0cm 0cm 0cm 6pt;margin-left:4.8pt;margin-right:0cm"> <p class=3D"MsoNormal">Am 17.04.24 um 05:00 schrieb Shu-Ju Tu:<br> > Hi dear Weka development team staff:<br> > <br> > I have a problem of getting low predictive accuracy when running a lar= ge <br> > data set.<br> > <br> > Here is the story and thank you for the patient in advance:<br> > We started a small data set (n=3D100) last year.<br> > It is a 2-class supervised data set and the class is evenly distribute= d <br> > 50-50.<br> > The correctly predictive accuracy on training after feature selection = <br> > and test data sets is about 85%.<br> > We have tried RandomForest and AdaBoostM1.<br> > Then we increased the data set to n=3D200 (later 300) and were getting= <br> > about similar predictive results.<br> > Then recently we increased to n=3D800 and were getting very low accura= cy <br> > of 60%.<br> > <br> > Are there something we can do and try to improve on the results?<br> <br> Maybe your new data is significantly different from the old data. If so, <b= r> you could try to retrain your model on the new data.<br> <br> I had a situation like that where I was looking at manufacturing data. <br> Then they reconfigured / optimised the machine and the data changed <br> enough to make my model useless.<br> <br> <br> _______________________________________________<br> Wekalist mailing list -- <a href=3D"mailto:[email protected]" tar= get=3D"_blank"> [email protected]</a><br> Send posts to <a href=3D"mailto:[email protected]" target=3D"_bla= nk">[email protected]</a><br> To unsubscribe send an email to <a href=3D"mailto:[email protected]= to.ac.nz" target=3D"_blank"> [email protected]</a><br> To subscribe, unsubscribe, etc., visit <a href=3D"https://list.waikato.ac.n= z/postorius/lists/wekalist.list.waikato.ac.nz" target=3D"_blank"> https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz</a><= br> List etiquette: <a href=3D"http://www.cs.waikato.ac.nz/~ml/weka/mailinglist= _etiquette.html" target=3D"_blank"> http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html</a><u></u><= u></u></p> </blockquote> </div> </div> </div> </div> </div> _______________________________________________<br> Wekalist mailing list -- <a href=3D"mailto:[email protected]" tar= get=3D"_blank">[email protected]</a><br> Send posts to <a href=3D"mailto:[email protected]" target=3D"_bla= nk">[email protected]</a><br> To unsubscribe send an email to <a href=3D"mailto:[email protected]= to.ac.nz" target=3D"_blank">[email protected]</a><br> To subscribe, unsubscribe, etc., visit <a href=3D"https://list.waikato.ac.n= z/postorius/lists/wekalist.list.waikato.ac.nz" rel=3D"noreferrer" target=3D= "_blank">https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.a= c.nz</a><br> List etiquette: <a href=3D"http://www.cs.waikato.ac.nz/~ml/weka/mailinglist= _etiquette.html" rel=3D"noreferrer" target=3D"_blank">http://www.cs.waikato= .ac.nz/~ml/weka/mailinglist_etiquette.html</a><br> </div></blockquote></div> --00000000000006fb450616688409-- --===============8361112038478535325== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Wekalist mailing list -- [email protected] Send posts to [email protected] To unsubscribe send an email to [email protected] To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html --===============8361112038478535325==--