Re: Problem of large data sets
Shu-Ju Tu <[email protected]> Wed, 24 Apr 2024 13:28:22 +0800
| Newsgroups | gmane.comp.ai.weka |
|---|---|
| Message-ID | <CABaQXBt4wBarC3JfftWSsSNFf6hVtyrfXMsz4veTVie5QZmAjQ@mail.gmail.com> |
--===============7327546181558797795== Content-Type: multipart/alternative; boundary="00000000000004055c0616d0f137" --00000000000004055c0616d0f137 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hello Frank, Thank you very much for taking time to answer my questions. I will take your advise and try AttributeSelectedClassifier. A quick question: Is there a standard way to check whether the distributions of older and newer data sets are similar or equivalent? Should I use statistical t-test to check for each attribute? Sincerely your and warmest regards, Shu-Ju Eibe Frank <[email protected]> =E6=96=BC 2024=E5=B9=B44=E6=9C=8820= =E6=97=A5 =E9=80=B1=E5=85=AD =E4=B8=8A=E5=8D=8810:50=E5=AF=AB=E9=81=93=EF= =BC=9A > Thanks for sharing the data. I don=E2=80=99t think any of the differences= I > observe are statistically significant. Classification accuracy always > hovers around 60%. > > > > You did not include any test set, so I assume you used 10-fold > cross-validation to estimate performance. Is it possible that you perform= ed > feature selection manually in the Preprocess panel (or similar) to achiev= e > 85% classification accuracy by tuning the attribute subset based on the > accuracy estimate obtained in the Classify panel? This could explain the > result: your smallest dataset has 70 instances and 121 attributes. It is > very likely that you will get optimistic performance estimates by tuning > the model based on the same dataset of 70 instances (and corresponding > evaluation process) that is used to establish the final accuracy estimate= . > > > > When you perform feature selection, it is essential to only use the > training data to do this. For example, if you use k-fold cross-validation > to establish the final accuracy estimate, then feature subsets must be > selected independently from each of the k training folds (e.g., by > performing nested k-fold cross-validation). The AttributeSelectedClassifi= er > will do this correctly for you. > > > > Cheers, > > Eibe > > > > *From: *Shu-Ju Tu <[email protected]> > *Date: *Friday, 19 April 2024 at 5:19=E2=80=AFPM > *To: *Weka machine learning workbench list. <[email protected]> > *Subject: *[Wekalist] Re: Problem of large data sets > > Hello Frank, > > > > Thank you very much for the quick response to our questions. > > I attached the following files of different sample sizes along with this > email. > > In the mean time, I will also try other options of classifiers as well. > > > > Sincerely yours and warm regards, > > Shu-Ju > > > > > > > > > > Eibe Frank <[email protected]> =E6=96=BC 2024=E5=B9=B44=E6=9C=8819= =E6=97=A5 =E9=80=B1=E4=BA=94 =E4=B8=8A=E5=8D=889:20=E5=AF=AB=E9=81=93=EF=BC= =9A > > Runtime will depend on the time complexity of the algorithm. RandomForest > and AdaBoostM1 (with decision stumps or REPTree as the base classifier) > scale quite well as you increase the number of samples: if N is the numbe= r > of samples, then the expected runtime should be O(N \times log N)). > > > > With fixed hyperparameter settings, the predictive performance of a > well-behaved learning algorithm should not drop (significantly) as you > increase the amount of training data (keeping the set of attributes the > same). > > > > I=E2=80=99d be happy to take a quick look at those datasets with WEKA if = you are > able to share them with me. > > > > Cheers, > > Eibe > > > > *From: *Shu-Ju Tu <[email protected]> > *Date: *Friday, 19 April 2024 at 12:54=E2=80=AFPM > *To: *Weka machine learning workbench list. <[email protected]> > *Subject: *[Wekalist] Re: Problem of large data sets > > Thank you very much for the reply to our problem. > > Yes and the difference is in the sample size of training data sets. > > #1 sample n=3D100 > > #2 sample n=3D200 > > #3 sample n=3D300 > > #4 sample n>900 > > > > Two quick follow-up questions: > > > > (1) A large data set will take a lot longer computing time of going > through the feature selection process correct? > > (2) For a large data set, do I need to have any special consideration of > parameter settings concerning Random Forest/AdaBoostM1 learning algorithm= s? > > > > Best regards, > > Shu-Ju > > > > > > > > > > > > > > > > > > > > Eibe Frank <[email protected]> =E6=96=BC 2024=E5=B9=B44=E6=9C=8819= =E6=97=A5 =E9=80=B1=E4=BA=94 =E4=B8=8A=E5=8D=887:05=E5=AF=AB=E9=81=93=EF=BC= =9A > > I assume you use the same test data in all three cases and only the > training data changes. For IID data, this kind of behaviour should not > happen with well-behaved learning algorithms such as RandomForest and > AdaBoostM1. > > > > Cheers, > > Eibe > > > > *From: *Shu-Ju Tu <[email protected]> > *Date: *Thursday, 18 April 2024 at 8:01=E2=80=AFPM > *To: *Weka machine learning workbench list. <[email protected]> > *Subject: *[Wekalist] Re: Problem of large data sets > > Hello Thank you very much for sharing your information. > > > > Our data were obtained from the same PET imaging machine and identical > settings. > > I believe our medical physicists routinely perform QA of high quality for > this machine. > > > > Previously I was thinking that is the problem of a large number of data > set (n>800). > > So that large number of data set (> 800) actually was not an issue? > > > > Shu-Ju > > > > Ulrich Mayring <[email protected]> =E6=96=BC 2024=E5=B9=B44=E6=9C=88= 18=E6=97=A5 =E9=80=B1=E5=9B=9B =E4=B8=8A=E5=8D=888:59=E5=AF=AB=E9=81=93=EF= =BC=9A > > Am 17.04.24 um 05:00 schrieb Shu-Ju Tu: > > Hi dear Weka development team staff: > > > > I have a problem of getting low predictive accuracy when running a larg= e > > data set. > > > > Here is the story and thank you for the patient in advance: > > We started a small data set (n=3D100) last year. > > It is a 2-class supervised data set and the class is evenly distributed > > 50-50. > > The correctly predictive accuracy on training after feature selection > > and test data sets is about 85%. > > We have tried RandomForest and AdaBoostM1. > > Then we increased the data set to n=3D200 (later 300) and were getting > > about similar predictive results. > > Then recently we increased to n=3D800 and were getting very low accurac= y > > of 60%. > > > > Are there something we can do and try to improve on the results? > > Maybe your new data is significantly different from the old data. If so, > you could try to retrain your model on the new data. > > I had a situation like that where I was looking at manufacturing data. > Then they reconfigured / optimised the machine and the data changed > enough to make my model useless. > > > _______________________________________________ > Wekalist mailing list -- [email protected] > Send posts to [email protected] > To unsubscribe send an email to [email protected] > To subscribe, unsubscribe, etc., visit > https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz > List etiquette: > http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html > > _______________________________________________ > Wekalist mailing list -- [email protected] > Send posts to [email protected] > To unsubscribe send an email to [email protected] > To subscribe, unsubscribe, etc., visit > https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz > List etiquette: > http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html > > _______________________________________________ > Wekalist mailing list -- [email protected] > Send posts to [email protected] > To unsubscribe send an email to [email protected] > To subscribe, unsubscribe, etc., visit > https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz > List etiquette: > http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html > > _______________________________________________ > Wekalist mailing list -- [email protected] > Send posts to [email protected] > To unsubscribe send an email to [email protected] > To subscribe, unsubscribe, etc., visit > https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz > List etiquette: > http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html > --00000000000004055c0616d0f137 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div>Hello Frank,</div><div><br></div><div>Thank you very = much for taking time to answer my questions.</div><div>I will take your adv= ise and try=20 <span style=3D"font-size:11pt">AttributeSelectedClassifier.</span></div><di= v><span style=3D"font-size:11pt"><br></span></div><div><span style=3D"font-= size:11pt">A quick question:</span></div><div><span style=3D"font-size:11pt= ">Is there a standard way to check whether the distributions of older and n= ewer data sets are similar or equivalent?</span></div><div><span style=3D"f= ont-size:11pt">Should I use statistical t-test to check for each attribute?= =C2=A0 <br></span></div><div><span style=3D"font-size:11pt"><br></span></di= v><div><span style=3D"font-size:11pt">Sincerely your and warmest regards,<b= r></span></div><div><span style=3D"font-size:11pt">Shu-Ju<br></span></div><= div><span style=3D"font-size:11pt"><br></span></div><div><span style=3D"fon= t-size:11pt"><br></span> </div><div><br></div><div><br></div></div><br><div class=3D"gmail_quote"><d= iv dir=3D"ltr" class=3D"gmail_attr">Eibe Frank <<a href=3D"mailto:eibe.f= [email protected]">[email protected]</a>> =E6=96=BC 2024=E5=B9= =B44=E6=9C=8820=E6=97=A5 =E9=80=B1=E5=85=AD =E4=B8=8A=E5=8D=8810:50=E5=AF= =AB=E9=81=93=EF=BC=9A<br></div><blockquote class=3D"gmail_quote" style=3D"m= argin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left= :1ex"><div class=3D"msg-7962366431958451221"> <div lang=3D"EN-NZ" style=3D"overflow-wrap: break-word;"> <div class=3D"m_-7962366431958451221WordSection1"> <p class=3D"MsoNormal"><span style=3D"font-size:11pt">Thanks for sharing th= e data. I don=E2=80=99t think any of the differences I observe are statisti= cally significant. Classification accuracy always hovers around 60%. <u></u><u></u></span></p> <p class=3D"MsoNormal"><span style=3D"font-size:11pt"><u></u>=C2=A0<u></u><= /span></p> <p class=3D"MsoNormal"><span style=3D"font-size:11pt">You did not include a= ny test set, so I assume you used 10-fold cross-validation to estimate perf= ormance. Is it possible that you performed feature selection manually in th= e Preprocess panel (or similar) to achieve 85% classification accuracy by tuning the at= tribute subset based on the accuracy estimate obtained in the Classify pane= l? This could explain the result: your smallest dataset has 70 instances an= d 121 attributes. It is very likely that you will get optimistic performance estimates by tuning the model bas= ed on the same dataset of 70 instances (and corresponding evaluation proces= s) that is used to establish the final accuracy estimate.<u></u><u></u></sp= an></p> <p class=3D"MsoNormal"><span style=3D"font-size:11pt"><u></u>=C2=A0<u></u><= /span></p> <p class=3D"MsoNormal"><span style=3D"font-size:11pt">When you perform feat= ure selection, it is essential to only use the training data to do this. Fo= r example, if you use k-fold cross-validation to establish the final accura= cy estimate, then feature subsets must be selected independently from each of the k tra= ining folds (e.g., by performing nested k-fold cross-validation). The Attri= buteSelectedClassifier will do this correctly for you.<u></u><u></u></span>= </p> <p class=3D"MsoNormal"><span style=3D"font-size:11pt"><u></u>=C2=A0<u></u><= /span></p> <p class=3D"MsoNormal"><span style=3D"font-size:11pt">Cheers,<u></u><u></u>= </span></p> <p class=3D"MsoNormal"><span style=3D"font-size:11pt">Eibe<u></u><u></u></s= pan></p> <p class=3D"MsoNormal"><span style=3D"font-size:11pt"><u></u>=C2=A0<u></u><= /span></p> <div id=3D"m_-7962366431958451221mail-editor-reference-message-container"> <div> <div style=3D"border-width:1pt medium medium;border-style:solid none none;b= order-color:rgb(181,196,223) currentcolor currentcolor;padding:3pt 0cm 0cm"= > <p class=3D"MsoNormal" style=3D"margin-bottom:12pt"><b><span style=3D"color= :black">From: </span></b><span style=3D"color:black">Shu-Ju Tu <<a href=3D"mailto:sjtu= @gap.cgu.edu.tw" target=3D"_blank">[email protected]</a>><br> <b>Date: </b>Friday, 19 April 2024 at 5:19</span><span style=3D"font-family= :"Arial",sans-serif;color:black">=E2=80=AF</span><span style=3D"c= olor:black">PM<br> <b>To: </b>Weka machine learning workbench list. <<a href=3D"mailto:weka= [email protected]" target=3D"_blank">[email protected]</a>&= gt;<br> <b>Subject: </b>[Wekalist] Re: Problem of large data sets<u></u><u></u></sp= an></p> </div> <div> <div> <p class=3D"MsoNormal">Hello Frank,<u></u><u></u></p> </div> <div> <p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p> </div> <div> <p class=3D"MsoNormal">Thank you very much for the quick response to our qu= estions.<u></u><u></u></p> </div> <div> <p class=3D"MsoNormal">I attached the following files of different sample s= izes along with this email.<u></u><u></u></p> </div> <div> <p class=3D"MsoNormal">In the mean time, I will also try other options of c= lassifiers as well.<u></u><u></u></p> </div> <div> <p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p> </div> <div> <p class=3D"MsoNormal">Sincerely yours and warm regards,<u></u><u></u></p> </div> <div> <p class=3D"MsoNormal">Shu-Ju<u></u><u></u></p> </div> <div> <p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p> </div> <div> <p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p> </div> <div> <p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p> </div> </div> <p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p> <div> <div> <p class=3D"MsoNormal">Eibe Frank <<a href=3D"mailto:eibe.frank@waikato.= ac.nz" target=3D"_blank">[email protected]</a>> <span lang=3D"EN-GB" style=3D"font-family:"MS Gothic"">=E6=96=BC<= /span> 2024<span lang=3D"EN-GB" style=3D"font-family:"MS Gothic""= >=E5=B9=B4</span>4<span lang=3D"EN-GB" style=3D"font-family:"MS Gothic= "">=E6=9C=88</span>19<span lang=3D"EN-GB" style=3D"font-family:"M= S Gothic"">=E6=97=A5</span><span lang=3D"EN-GB"> </span><span lang=3D"EN-GB" style=3D"font-family:"MS Gothic"">=E9= =80=B1=E4=BA=94</span><span lang=3D"EN-GB"> </span><span lang=3D"EN-GB" style=3D"font-family:"MS Gothic"">=E4= =B8=8A=E5=8D=88</span>9:20<span lang=3D"EN-GB" style=3D"font-family:"M= S Gothic"">=E5=AF=AB=E9=81=93=EF=BC=9A</span><u></u><u></u></p> </div> <blockquote style=3D"border-width:medium medium medium 1pt;border-style:non= e none none solid;border-color:currentcolor currentcolor currentcolor rgb(2= 04,204,204);padding:0cm 0cm 0cm 6pt;margin-left:4.8pt;margin-right:0cm"> <div> <div> <div> <p class=3D"MsoNormal"><span style=3D"font-size:11pt">Runtime will depend o= n the time complexity of the algorithm. RandomForest and AdaBoostM1 (with d= ecision stumps or REPTree as the base classifier) scale quite well as you increase the number of samples: if N is the number= of samples, then the expected runtime should be O(N \times log N)).</span>= <u></u><u></u></p> <p class=3D"MsoNormal"><span style=3D"font-size:11pt">=C2=A0</span><u></u><= u></u></p> <p class=3D"MsoNormal"><span style=3D"font-size:11pt">With fixed hyperparam= eter settings, the predictive performance of a well-behaved learning algori= thm should not drop (significantly) as you increase the amount of training data (keeping the set of attributes the same).</spa= n><u></u><u></u></p> <p class=3D"MsoNormal"><span style=3D"font-size:11pt">=C2=A0</span><u></u><= u></u></p> <p class=3D"MsoNormal"><span style=3D"font-size:11pt">I=E2=80=99d be happy = to take a quick look at those datasets with WEKA if you are able to share t= hem with me.</span><u></u><u></u></p> <p class=3D"MsoNormal"><span style=3D"font-size:11pt">=C2=A0</span><u></u><= u></u></p> <p class=3D"MsoNormal"><span style=3D"font-size:11pt">Cheers,</span><u></u>= <u></u></p> <p class=3D"MsoNormal"><span style=3D"font-size:11pt">Eibe</span><u></u><u>= </u></p> <p class=3D"MsoNormal"><span style=3D"font-size:11pt">=C2=A0</span><u></u><= u></u></p> <div id=3D"m_-7962366431958451221m_-3684511803295880724mail-editor-referenc= e-message-container"> <div> <div style=3D"border-width:1pt medium medium;border-style:solid none none;p= adding:3pt 0cm 0cm;border-color:currentcolor"> <p class=3D"MsoNormal" style=3D"margin-bottom:12pt"><b><span style=3D"color= :black">From: </span></b><span style=3D"color:black">Shu-Ju Tu <<a href=3D"mailto:sjtu= @gap.cgu.edu.tw" target=3D"_blank">[email protected]</a>><br> <b>Date: </b>Friday, 19 April 2024 at 12:54</span><span style=3D"font-famil= y:"Arial",sans-serif;color:black">=E2=80=AF</span><span style=3D"= color:black">PM<br> <b>To: </b>Weka machine learning workbench list. <<a href=3D"mailto:weka= [email protected]" target=3D"_blank">[email protected]</a>&= gt;<br> <b>Subject: </b>[Wekalist] Re: Problem of large data sets</span><u></u><u><= /u></p> </div> <div> <div> <p class=3D"MsoNormal">Thank you very much for the reply to our problem.<u>= </u><u></u></p> </div> <div> <p class=3D"MsoNormal">Yes and the difference is in the sample size of trai= ning data sets.<u></u><u></u></p> </div> <div> <p class=3D"MsoNormal">#1 sample n=3D100<u></u><u></u></p> </div> <div> <p class=3D"MsoNormal">#2 sample n=3D200<u></u><u></u></p> </div> <div> <p class=3D"MsoNormal">#3 sample n=3D300<u></u><u></u></p> </div> <div> <p class=3D"MsoNormal">#4 sample n>900<u></u><u></u></p> </div> <div> <p class=3D"MsoNormal">=C2=A0<u></u><u></u></p> </div> <div> <p class=3D"MsoNormal">Two quick follow-up questions:<u></u><u></u></p> </div> <div> <p class=3D"MsoNormal">=C2=A0<u></u><u></u></p> </div> <div> <p class=3D"MsoNormal">(1) A large data set will take a lot longer computin= g time of going through the feature selection process correct?<u></u><u></u= ></p> </div> <div> <p class=3D"MsoNormal">(2) For a large data set, do I need to have any spec= ial consideration of parameter settings concerning Random Forest/AdaBoostM1= learning algorithms?<u></u><u></u></p> </div> <div> <p class=3D"MsoNormal">=C2=A0<u></u><u></u></p> </div> <div> <p class=3D"MsoNormal">Best regards,<u></u><u></u></p> </div> <div> <p class=3D"MsoNormal">Shu-Ju<u></u><u></u></p> </div> <div> <p class=3D"MsoNormal">=C2=A0<u></u><u></u></p> </div> <div> <p class=3D"MsoNormal">=C2=A0<u></u><u></u></p> </div> <div> <p class=3D"MsoNormal">=C2=A0<u></u><u></u></p> </div> <div> <p class=3D"MsoNormal">=C2=A0<u></u><u></u></p> </div> <div> <p class=3D"MsoNormal">=C2=A0<u></u><u></u></p> </div> <div> <p class=3D"MsoNormal">=C2=A0<u></u><u></u></p> </div> <div> <p class=3D"MsoNormal">=C2=A0<u></u><u></u></p> </div> <div> <p class=3D"MsoNormal">=C2=A0<u></u><u></u></p> </div> </div> <p class=3D"MsoNormal">=C2=A0<u></u><u></u></p> <div> <div> <p class=3D"MsoNormal">Eibe Frank <<a href=3D"mailto:eibe.frank@waikato.= ac.nz" target=3D"_blank">[email protected]</a>> <span lang=3D"EN-GB" style=3D"font-family:"MS Gothic"">=E6=96=BC<= /span> 2024<span lang=3D"EN-GB" style=3D"font-family:"MS Gothic""= >=E5=B9=B4</span>4<span lang=3D"EN-GB" style=3D"font-family:"MS Gothic= "">=E6=9C=88</span>19<span lang=3D"EN-GB" style=3D"font-family:"M= S Gothic"">=E6=97=A5</span><span lang=3D"EN-GB"> </span><span lang=3D"EN-GB" style=3D"font-family:"MS Gothic"">=E9= =80=B1=E4=BA=94</span><span lang=3D"EN-GB"> </span><span lang=3D"EN-GB" style=3D"font-family:"MS Gothic"">=E4= =B8=8A=E5=8D=88</span>7:05<span lang=3D"EN-GB" style=3D"font-family:"M= S Gothic"">=E5=AF=AB=E9=81=93=EF=BC=9A</span><u></u><u></u></p> </div> <blockquote style=3D"border-width:medium medium medium 1pt;border-style:non= e none none solid;padding:0cm 0cm 0cm 6pt;margin:5pt 0cm 5pt 4.8pt;border-c= olor:currentcolor currentcolor currentcolor rgb(204,204,204)"> <div> <div> <div> <p class=3D"MsoNormal"><span style=3D"font-size:11pt">I assume you use the = same test data in all three cases and only the training data changes. For I= ID data, this kind of behaviour should not happen with well-behaved learning algorithms such as RandomForest and AdaBoostM1.= </span><u></u><u></u></p> <p class=3D"MsoNormal"><span style=3D"font-size:11pt">=C2=A0</span><u></u><= u></u></p> <p class=3D"MsoNormal"><span style=3D"font-size:11pt">Cheers,</span><u></u>= <u></u></p> <p class=3D"MsoNormal"><span style=3D"font-size:11pt">Eibe</span><u></u><u>= </u></p> <p class=3D"MsoNormal"><span style=3D"font-size:11pt">=C2=A0</span><u></u><= u></u></p> <div id=3D"m_-7962366431958451221m_-3684511803295880724m_-69097374691206225= 09mail-editor-reference-message-container"> <div> <div style=3D"border-width:1pt medium medium;border-style:solid none none;p= adding:3pt 0cm 0cm;border-color:currentcolor"> <p class=3D"MsoNormal" style=3D"margin-bottom:12pt"><b><span style=3D"color= :black">From: </span></b><span style=3D"color:black">Shu-Ju Tu <<a href=3D"mailto:sjtu= @gap.cgu.edu.tw" target=3D"_blank">[email protected]</a>><br> <b>Date: </b>Thursday, 18 April 2024 at 8:01</span><span style=3D"font-fami= ly:"Arial",sans-serif;color:black">=E2=80=AF</span><span style=3D= "color:black">PM<br> <b>To: </b>Weka machine learning workbench list. <<a href=3D"mailto:weka= [email protected]" target=3D"_blank">[email protected]</a>&= gt;<br> <b>Subject: </b>[Wekalist] Re: Problem of large data sets</span><u></u><u><= /u></p> </div> <div> <div> <p class=3D"MsoNormal">Hello Thank you very much for sharing your informati= on.<u></u><u></u></p> </div> <div> <p class=3D"MsoNormal">=C2=A0<u></u><u></u></p> </div> <div> <p class=3D"MsoNormal">Our data were obtained from the same PET imaging mac= hine and identical settings.<u></u><u></u></p> </div> <div> <p class=3D"MsoNormal">I believe our medical physicists routinely perform Q= A of high quality for this machine.<u></u><u></u></p> </div> <div> <p class=3D"MsoNormal">=C2=A0<u></u><u></u></p> </div> <div> <p class=3D"MsoNormal">Previously I was thinking that is the problem of a l= arge number of data set (n>800).<u></u><u></u></p> </div> <div> <p class=3D"MsoNormal">So that large number of data set (> 800) actually= was not an issue?<u></u><u></u></p> </div> <div> <p class=3D"MsoNormal">=C2=A0<u></u><u></u></p> </div> <div> <p class=3D"MsoNormal">Shu-Ju<u></u><u></u></p> </div> </div> <p class=3D"MsoNormal">=C2=A0<u></u><u></u></p> <div> <div> <p class=3D"MsoNormal">Ulrich Mayring <<a href=3D"mailto:ulrich.mayring@= isys.de" target=3D"_blank">[email protected]</a>> <span lang=3D"EN-GB" style=3D"font-family:"MS Gothic"">=E6=96=BC<= /span> 2024<span lang=3D"EN-GB" style=3D"font-family:"MS Gothic""= >=E5=B9=B4</span>4<span lang=3D"EN-GB" style=3D"font-family:"MS Gothic= "">=E6=9C=88</span>18<span lang=3D"EN-GB" style=3D"font-family:"M= S Gothic"">=E6=97=A5</span><span lang=3D"EN-GB"> </span><span lang=3D"EN-GB" style=3D"font-family:"MS Gothic"">=E9= =80=B1=E5=9B=9B</span><span lang=3D"EN-GB"> </span><span lang=3D"EN-GB" style=3D"font-family:"MS Gothic"">=E4= =B8=8A=E5=8D=88</span>8:59<span lang=3D"EN-GB" style=3D"font-family:"M= S Gothic"">=E5=AF=AB=E9=81=93=EF=BC=9A</span><u></u><u></u></p> </div> <blockquote style=3D"border-width:medium medium medium 1pt;border-style:non= e none none solid;padding:0cm 0cm 0cm 6pt;margin:5pt 0cm 5pt 4.8pt;border-c= olor:currentcolor currentcolor currentcolor rgb(204,204,204)"> <p class=3D"MsoNormal">Am 17.04.24 um 05:00 schrieb Shu-Ju Tu:<br> > Hi dear Weka development team staff:<br> > <br> > I have a problem of getting low predictive accuracy when running a lar= ge <br> > data set.<br> > <br> > Here is the story and thank you for the patient in advance:<br> > We started a small data set (n=3D100) last year.<br> > It is a 2-class supervised data set and the class is evenly distribute= d <br> > 50-50.<br> > The correctly predictive accuracy on training after feature selection = <br> > and test data sets is about 85%.<br> > We have tried RandomForest and AdaBoostM1.<br> > Then we increased the data set to n=3D200 (later 300) and were getting= <br> > about similar predictive results.<br> > Then recently we increased to n=3D800 and were getting very low accura= cy <br> > of 60%.<br> > <br> > Are there something we can do and try to improve on the results?<br> <br> Maybe your new data is significantly different from the old data. If so, <b= r> you could try to retrain your model on the new data.<br> <br> I had a situation like that where I was looking at manufacturing data. <br> Then they reconfigured / optimised the machine and the data changed <br> enough to make my model useless.<br> <br> <br> _______________________________________________<br> Wekalist mailing list -- <a href=3D"mailto:[email protected]" tar= get=3D"_blank"> [email protected]</a><br> Send posts to <a href=3D"mailto:[email protected]" target=3D"_bla= nk">[email protected]</a><br> To unsubscribe send an email to <a href=3D"mailto:[email protected]= to.ac.nz" target=3D"_blank"> [email protected]</a><br> To subscribe, unsubscribe, etc., visit <a href=3D"https://list.waikato.ac.n= z/postorius/lists/wekalist.list.waikato.ac.nz" target=3D"_blank"> https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz</a><= br> List etiquette: <a href=3D"http://www.cs.waikato.ac.nz/~ml/weka/mailinglist= _etiquette.html" target=3D"_blank"> http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html</a><u></u><= u></u></p> </blockquote> </div> </div> </div> </div> </div> <p class=3D"MsoNormal">_______________________________________________<br> Wekalist mailing list -- <a href=3D"mailto:[email protected]" tar= get=3D"_blank"> [email protected]</a><br> Send posts to <a href=3D"mailto:[email protected]" target=3D"_bla= nk">[email protected]</a><br> To unsubscribe send an email to <a href=3D"mailto:[email protected]= to.ac.nz" target=3D"_blank"> [email protected]</a><br> To subscribe, unsubscribe, etc., visit <a href=3D"https://list.waikato.ac.n= z/postorius/lists/wekalist.list.waikato.ac.nz" target=3D"_blank"> https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz</a><= br> List etiquette: <a href=3D"http://www.cs.waikato.ac.nz/~ml/weka/mailinglist= _etiquette.html" target=3D"_blank"> http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html</a><u></u><= u></u></p> </div> </blockquote> </div> </div> </div> </div> </div> <p class=3D"MsoNormal">_______________________________________________<br> Wekalist mailing list -- <a href=3D"mailto:[email protected]" tar= get=3D"_blank"> [email protected]</a><br> Send posts to <a href=3D"mailto:[email protected]" target=3D"_bla= nk">[email protected]</a><br> To unsubscribe send an email to <a href=3D"mailto:[email protected]= to.ac.nz" target=3D"_blank"> [email protected]</a><br> To subscribe, unsubscribe, etc., visit <a href=3D"https://list.waikato.ac.n= z/postorius/lists/wekalist.list.waikato.ac.nz" target=3D"_blank"> https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz</a><= br> List etiquette: <a href=3D"http://www.cs.waikato.ac.nz/~ml/weka/mailinglist= _etiquette.html" target=3D"_blank"> http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html</a><u></u><= u></u></p> </div> </blockquote> </div> </div> </div> </div> </div> _______________________________________________<br> Wekalist mailing list -- <a href=3D"mailto:[email protected]" tar= get=3D"_blank">[email protected]</a><br> Send posts to <a href=3D"mailto:[email protected]" target=3D"_bla= nk">[email protected]</a><br> To unsubscribe send an email to <a href=3D"mailto:[email protected]= to.ac.nz" target=3D"_blank">[email protected]</a><br> To subscribe, unsubscribe, etc., visit <a href=3D"https://list.waikato.ac.n= z/postorius/lists/wekalist.list.waikato.ac.nz" rel=3D"noreferrer" target=3D= "_blank">https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.a= c.nz</a><br> List etiquette: <a href=3D"http://www.cs.waikato.ac.nz/~ml/weka/mailinglist= _etiquette.html" rel=3D"noreferrer" target=3D"_blank">http://www.cs.waikato= .ac.nz/~ml/weka/mailinglist_etiquette.html</a><br> </div></blockquote></div> --00000000000004055c0616d0f137-- --===============7327546181558797795== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Wekalist mailing list -- [email protected] Send posts to [email protected] To unsubscribe send an email to [email protected] To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html --===============7327546181558797795==--