RE: performance bottleneck in yy_get_next_buffer function
Gautam Kapoor <[email protected]> Tue, 15 Jan 2013 14:27:24 +0530
| Newsgroups | gmane.comp.lex.flex.general |
|---|---|
| Message-ID | <664A5BEDF5FDDC4BA499CFCA7FCFAC990FDA7ABF60@MAILIN2.global.cadence.com> |
--_002_664A5BEDF5FDDC4BA499CFCA7FCFAC990FDA7ABF60MAILIN2global_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Hi Hans, This is true both for flex and flex++. Yes, my patch fixed that problem. Ideally, I would expect memcpy to be used= instead of a for-do there. But Martin (attached email) suggests that it co= uld be fixed by changing the buffer size. I have to try that. If one can ch= oose an optimal buffer size (may not always be possible), then the effect o= f this for-do will be minimized. -regards Gautam -----Original Message----- From: Hans =C5berg [mailto:[email protected]]=20 Sent: Thursday, January 10, 2013 5:00 AM To: Gautam Kapoor Cc: [email protected] Subject: Re: performance bottleneck in yy_get_next_buffer function On 9 Jan 2013, at 18:20, Gautam Kapoor <[email protected]> wrote: > I want to discuss a particular performance issue and how I tried to fix i= t. I am wondering why it is not part of the default scanner generated by fl= ex because I think the developers must have seen this too. Flex is currently not developed or maintained, it seems. But a known proble= m is that rules that capture a lot of input slows the lexer down. So, for e= xample, scanning for a comment spanning several lines is better done line b= y line. Does your patch fix that problem? Hans --_002_664A5BEDF5FDDC4BA499CFCA7FCFAC990FDA7ABF60MAILIN2global_ Content-Type: message/rfc822 Received: from mx-sanjose2.cadence.com (158.140.2.61) by exowa02in.global.cadence.com (172.23.38.150) with Microsoft SMTP Server (TLS) id 8.3.245.1; Thu, 10 Jan 2013 14:36:39 +0530 Received: from mx0a-0014ca01.pphosted.com ([208.84.65.235]) by mx-sanjose2.cadence.com (8.14.4/8.14.4) with ESMTP id r0A94xZl023547 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for <[email protected]>; Thu, 10 Jan 2013 01:04:59 -0800 (PST) Received: from pps.filterd (m0042385 [127.0.0.1]) by mx0a-0014ca01.pphosted.com (8.14.5/8.14.5) with SMTP id r0A96agV014442 for <[email protected]>; Thu, 10 Jan 2013 01:06:36 -0800 Received: from psmtp.com (exprod6mx206.postini.com [64.18.1.106]) by mx0a-0014ca01.pphosted.com with ESMTP id 19sf0uskb2-1 for <[email protected]>; Thu, 10 Jan 2013 01:06:36 -0800 Received: from snt0-omc1-s22.snt0.hotmail.com ([65.55.90.33]) by exprod6mx206.postini.com ([64.18.5.10]) with SMTP; Thu, 10 Jan 2013 09:06:36 GMT Received: from SNT127-W18 ([65.55.90.7]) by snt0-omc1-s22.snt0.hotmail.com with Microsoft SMTPSVC(6.0.3790.4675); Thu, 10 Jan 2013 01:06:35 -0800 From: "Martin.Beers Hotmail ." <[email protected]> To: Gautam Kapoor <[email protected]> Date: Thu, 10 Jan 2013 14:36:35 +0530 Subject: RE: performance bottleneck in yy_get_next_buffer function Thread-Topic: performance bottleneck in yy_get_next_buffer function Thread-Index: Ac3vEc1OWY4Cunl/SDyv6NV1i4wmCA== Message-ID: <[email protected]> References: <664A5BEDF5FDDC4BA499CFCA7FCFAC990FDA57BE01@MAILIN2.global.cadence.com> In-Reply-To: <664A5BEDF5FDDC4BA499CFCA7FCFAC990FDA57BE01@MAILIN2.global.cadence.com> X-MS-Exchange-Organization-AuthAs: Anonymous X-MS-Exchange-Organization-AuthSource: exowa02in.global.cadence.com X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originalarrivaltime: 10 Jan 2013 09:06:35.0770 (UTC) FILETIME=[CAB11DA0:01CDEF11] x-pstn-neptune: 0/0/0.00/0 x-received: by mx-sanjose2.cadence.com as r0A94xZl023547 at Thu Jan 10 01:04:59 2013 x-pstn-settings: 3 (1.0000:1.0000) s cv gt3 gt2 gt1 r p m c x-pstn-addresses: from <[email protected]> [129/6] x-pstn-levels: (S:23.23007/99.90000 CV:99.9000 FC:95.5390 LC:95.5390 R:95.9108 P:95.9108 M:88.1613 C:98.6951 ) x-originating-email: [[email protected]] x-proofpoint-virus-version: vendor=fsecure engine=2.50.10432:5.9.8327,1.0.431,0.0.0000 definitions=2013-01-10_02:2013-01-10,2013-01-09,1970-01-01 signatures=0 x-proofpoint-spam-details: rule=cadence_passive_policy_notspam policy=cadence_passive_policy score=0 spamscore=0 suspectscore=12 phishscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=7.0.1-1211240000 definitions=main-1301100017 x-pstn-dkim: 0 skipped:not-enabled Content-Type: multipart/alternative; boundary="_000_SNT127W187579A7615B822C218FFB8F2A0phxgbl_" MIME-Version: 1.0 --_000_SNT127W187579A7615B822C218FFB8F2A0phxgbl_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Hi Gautam, If your scanner spends a lot of time moving those buffers, then the buffer = is too small for your maximum token length. Both can be defined somewhere, = if I remember correctly. The buffer should be much much bigger then the maximum token length. regards, Martin > From: [email protected] > To: [email protected] > Date: Wed, 9 Jan 2013 22:50:15 +0530 > Subject: performance bottleneck in yy_get_next_buffer function > > Hi Compiler Experts, > I am using flex to generate a scanner. It creates a yy_get_next_buffer() = function in lex.yy.cc among other things. > > I want to discuss a particular performance issue and how I tried to fix i= t. I am wondering why it is not part of the default scanner generated by fl= ex because I think the developers must have seen this too. > > > When I run a particular test-case, I see a lot of time being taken by thi= s function (yy_get_next_buffer). On running tools like "quantify" to measur= e performance, I found that these particular lines in yy_get_next_buffer() = are taking a lot of time. > > /* First move last chars to start of buffer. */ > number_to_move =3D (int) ((yy_c_buf_p) - (yytext_ptr)) - 1; > > for ( i =3D 0; i < number_to_move; ++i ) > *(dest++) =3D *(source++); > > > When I change this to the following, I get a huge improvement in performa= nce. > > /* First move last chars to start of buffer. */ > number_to_move =3D (int) ((yy_c_buf_p) - (yytext_ptr)) - 1; > > memcpy(dest,source,number_to_move); > > > -regards > Gautam > > _______________________________________________ > help-flex mailing list > [email protected] > https://lists.gnu.org/mailman/listinfo/help-flex --_000_SNT127W187579A7615B822C218FFB8F2A0phxgbl_ Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable <html> <head> <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Diso-8859-= 1"> <style><!-- .hmmessage P { margin:0px; padding:0px } body.hmmessage { font-size: 10pt; font-family:Tahoma } --></style> </head> <body class=3D"hmmessage"> <div dir=3D"ltr"><br> <br> Hi Gautam,<br> <br> If your scanner spends a lot of time moving those buffers, then the buffer = is too small for your maximum token length. Both can be defined somewhere, = if I remember correctly.<br> <br> The buffer should be much much bigger then the maximum token length.<br> <br> <br> regards,<br> Martin<br> <br> <br> <div> <div id=3D"SkyDrivePlaceholder"></div> > From: [email protected]<br> > To: [email protected]<br> > Date: Wed, 9 Jan 2013 22:50:15 +0530<br> > Subject: performance bottleneck in yy_get_next_buffer function<br> > <br> > Hi Compiler Experts,<br> > I am using flex to generate a scanner. It creates a yy_get_next_buffer= () function in lex.yy.cc among other things.<br> > <br> > I want to discuss a particular performance issue and how I tried to fi= x it. I am wondering why it is not part of the default scanner generated by= flex because I think the developers must have seen this too.<br> > <br> > <br> > When I run a particular test-case, I see a lot of time being taken by = this function (yy_get_next_buffer). On running tools like "quantify&qu= ot; to measure performance, I found that these particular lines in yy_get_n= ext_buffer() are taking a lot of time.<br> > <br> > /* First move last chars to start of buffer. */<br> > number_to_move =3D (int) ((yy_c_buf_p) - (yytext_ptr)) - 1;<br> > <br> > for ( i =3D 0; i < number_to_move; ++i )<br> > *(dest++) =3D *(source++);<br> > <br> > <br> > When I change this to the following, I get a huge improvement in perfo= rmance. <br> > <br> > /* First move last chars to start of buffer. */<br> > number_to_move =3D (int) ((yy_c_buf_p) - (yytext_ptr)) - 1;<br> > <br> > memcpy(dest,source,number_to_move);<br> > <br> > <br> > -regards<br> > Gautam<br> > <br> > _______________________________________________<br> > help-flex mailing list<br> > [email protected]<br> > https://lists.gnu.org/mailman/listinfo/help-flex<br> </div> </div> </body> </html> --_000_SNT127W187579A7615B822C218FFB8F2A0phxgbl_-- --_002_664A5BEDF5FDDC4BA499CFCA7FCFAC990FDA7ABF60MAILIN2global_ Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ help-flex mailing list [email protected] https://lists.gnu.org/mailman/listinfo/help-flex --_002_664A5BEDF5FDDC4BA499CFCA7FCFAC990FDA7ABF60MAILIN2global_--