Tuning for throughput
<[email protected]> Tue, 2 Apr 2013 14:32:50 +0000
| Newsgroups | gmane.mail.spam.crm114 |
|---|---|
| Message-ID | <[email protected]> |
--===============0079961640582029998== Content-Language: en-US Content-Type: multipart/alternative; boundary="_000_163710A0BBE76645B2D0E92FDA1A5D2A05FED2MIVEXAMER1N2corpn_" --_000_163710A0BBE76645B2D0E92FDA1A5D2A05FED2MIVEXAMER1N2corpn_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Greetings, denizens of the list. I am investigating using libcrm114 for mass classification and I need to un= derstand the run-time space and time characteristics. Has anyone else looke= d at this aspect? I have running test cases with different volumes of mail that has already b= een classified. What I see is that the time to classify a message increases= in a generally linear fashion as the size of the classifier grows until I = get a "large" classifier (15,000 messages or more) where the time required = kicks up sharply. I am guessing that this is due to memory requirements an= d swapping starts to occur. Other than the process memory requirements, are there any other known issue= s with libcrm114 that make its performance non-linear for larger volumes of= mail? If it is using a hash lookup then I expect see linear performance o= n insertion and lookup. Thanks, Martin Martin Thomas Research Tools Software Architect Risk and Compliance Security Research McAfee, Inc. 972.963.7680 Direct [email protected]<mailto:[email protected]> --_000_163710A0BBE76645B2D0E92FDA1A5D2A05FED2MIVEXAMER1N2corpn_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable <html xmlns:v=3D"urn:schemas-microsoft-com:vml" xmlns:o=3D"urn:schemas-micr= osoft-com:office:office" xmlns:w=3D"urn:schemas-microsoft-com:office:word" = xmlns:m=3D"http://schemas.microsoft.com/office/2004/12/omml" xmlns=3D"http:= //www.w3.org/TR/REC-html40"> <head> <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dus-ascii"= > <meta name=3D"Generator" content=3D"Microsoft Word 12 (filtered medium)"> <style><!-- /* Font Definitions */ @font-face {font-family:Helvetica; panose-1:2 11 6 4 2 2 2 2 2 4;} @font-face {font-family:Helvetica; panose-1:2 11 6 4 2 2 2 2 2 4;} @font-face {font-family:Calibri; panose-1:2 15 5 2 2 2 4 3 2 4;} /* Style Definitions */ p.MsoNormal, li.MsoNormal, div.MsoNormal {margin:0in; margin-bottom:.0001pt; font-size:11.0pt; font-family:"Calibri","sans-serif";} a:link, span.MsoHyperlink {mso-style-priority:99; color:blue; text-decoration:underline;} a:visited, span.MsoHyperlinkFollowed {mso-style-priority:99; color:purple; text-decoration:underline;} span.EmailStyle17 {mso-style-type:personal-compose; font-family:"Calibri","sans-serif"; color:windowtext;} .MsoChpDefault {mso-style-type:export-only;} @page WordSection1 {size:8.5in 11.0in; margin:1.0in 1.0in 1.0in 1.0in;} div.WordSection1 {page:WordSection1;} --></style><!--[if gte mso 9]><xml> <o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" /> </xml><![endif]--><!--[if gte mso 9]><xml> <o:shapelayout v:ext=3D"edit"> <o:idmap v:ext=3D"edit" data=3D"1" /> </o:shapelayout></xml><![endif]--> </head> <body lang=3D"EN-US" link=3D"blue" vlink=3D"purple"> <div class=3D"WordSection1"> <p class=3D"MsoNormal">Greetings, denizens of the list.<o:p></o:p></p> <p class=3D"MsoNormal">I am investigating using libcrm114 for mass classifi= cation and I need to understand the run-time space and time characteristics= . Has anyone else looked at this aspect?<o:p></o:p></p> <p class=3D"MsoNormal"><o:p> </o:p></p> <p class=3D"MsoNormal">I have running test cases with different volumes of = mail that has already been classified. What I see is that the time to class= ify a message increases in a generally linear fashion as the size of the cl= assifier grows until I get a “large” classifier (15,000 messages or more) where the time required kicks up shar= ply. I am guessing that this is due to memory requirements and swappi= ng starts to occur.<o:p></o:p></p> <p class=3D"MsoNormal"><o:p> </o:p></p> <p class=3D"MsoNormal">Other than the process memory requirements, are ther= e any other known issues with libcrm114 that make its performance non-linea= r for larger volumes of mail? If it is using a hash lookup then I exp= ect see linear performance on insertion and lookup.<o:p></o:p></p> <p class=3D"MsoNormal"><o:p> </o:p></p> <p class=3D"MsoNormal">Thanks,<o:p></o:p></p> <p class=3D"MsoNormal">Martin<o:p></o:p></p> <p class=3D"MsoNormal"><o:p> </o:p></p> <p class=3D"MsoNormal"><o:p> </o:p></p> <p class=3D"MsoNormal"><b><span style=3D"font-size:10.0pt;color:black">Mart= in Thomas</span></b><b><span style=3D"font-size:10.0pt;color:black"><br> </span></b><span style=3D"font-size:10.0pt;color:black"><br> </span><span style=3D"font-size:10.0pt;color:black">Research Tools Software= Architect</span><span style=3D"font-size:10.0pt;color:black"><br> </span><span style=3D"font-size:10.0pt;color:black">Risk and Compliance Sec= urity Research</span><span style=3D"font-size:10.0pt;color:black"><br> </span><span style=3D"font-size:10.0pt;color:black">McAfee, Inc.</span><spa= n style=3D"font-size:10.0pt;color:black"><br> <br> </span><span style=3D"font-size:10.0pt;color:black">972.963.7680 Direct</sp= an><span style=3D"font-size:10.0pt;color:black"><o:p></o:p></span></p> <p class=3D"MsoNormal"><span style=3D"font-size:10.0pt;color:black"><br> </span><span style=3D"font-size:10.0pt;color:black"><a href=3D"mailto:marti= [email protected]"><span style=3D"color:blue">[email protected]</s= pan></a><o:p></o:p></span></p> <p class=3D"MsoNormal"><o:p> </o:p></p> </div> </body> </html> --_000_163710A0BBE76645B2D0E92FDA1A5D2A05FED2MIVEXAMER1N2corpn_-- --===============0079961640582029998== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline ------------------------------------------------------------------------------ Own the Future-Intel(R) Level Up Game Demo Contest 2013 Rise to greatness in Intel's independent game demo contest. Compete for recognition, cash, and the chance to get your game on Steam. $5K grand prize plus 10 genre and skill prizes. Submit your demo by 6/6/13. http://altfarm.mediaplex.com/ad/ck/12124-176961-30367-2 --===============0079961640582029998== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Crm114-general mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/crm114-general --===============0079961640582029998==--