C Code Parser Using Recursive Descent
[email protected] (Rahul Jain) Fri, 9 Oct 2009 05:12:23 -0700 (PDT)
| Newsgroups | perl.recdescent |
|---|---|
| Message-ID | <[email protected]> |
--0-2040507781-1255090343=:10862
Content-Type: multipart/alternative; boundary="0-1332330031-1255090343=:10862"
--0-1332330031-1255090343=:10862
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable
Hi All,
=A0
I am working on a C code parser, in which one of my requirements is to pars=
e the C source and header files and calculate the Lines of Code. Though the=
re are tools to do so, all of them have a problem wherein they treat functi=
on definitions and declarations=A0with argument list on multiple lines as m=
ultiple lines rather than single line . For example=20
=A0
- 1-
=A0
/* Definition */
int Test ( int x,
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 int y
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 )
{
}
=A0
or
=A0
/* Declaration */=A0
int Test( int x,
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 int y
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 );
=A0
is treated as=A05 and 3 lines respectively rather than 3 and 1 line, i.e it=
should be treated as
=A0
-=A02 -=A0=A0
/* Definition */
int Test ( int x, int y)
{
}
=A0
/* Declaration */=A0
int Test( int x, int y );
=A0
To fix this I=A0plan to make some modifications in my perl tool. I wish to =
use RecDescent to parse the input file, identify such constucts=A0and then =
use perl script to convert these=A0multiple line construct into single line=
contructs, so if construct=A0- 1 -=A0is given as input to the script then =
the output should be - 2=A0-. I found a script by Damian Conway, Helmut Jar=
ausch and Teodor Zlatanov which uses the RecDescent to seperate comments fr=
om the c code.(also attached with the mail)=A0The grammar used is=20
=A0
C_code : m{(=20
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 =A0[^"/]+=A0=
=A0=A0=A0=A0 # one or more non-delimiters
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 (=A0=A0=A0=
=A0=A0=A0=A0=A0=A0=A0=A0 # then (optionally)...
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0/=A0=A0=
=A0=A0=A0=A0=A0=A0=A0=A0=A0 # a potential comment delimiter
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 [^*/]=A0=A0=
=A0=A0=A0=A0 # which is not an actual delimiter
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0)?=A0=A0=
=A0=A0=A0=A0=A0=A0=A0 #=20
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 )+=A0=A0=A0=A0=
=A0=A0=A0=A0=A0 # all repeated once or more
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 }x
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 { $Code .=3D $item[1] }
=A0
comment : m{=A0=A0 \s*=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 =A0# optional wh=
itespace
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 //=A0=A0=
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 =A0# comment delimiter
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 [^\n]*=
=A0=A0=A0=A0=A0=A0=A0=A0 =A0 # anything except a newline
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 \n=A0=A0=
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 # then a newline
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 }x
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 {=A0 $Code .=3D "=
\n"; $Comments .=3D $item[1] }
=A0
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 |=A0=A0 m{\s*=A0=
=A0=A0=A0 =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 =A0# optional whitespace
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=
/\*=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 =A0# comment =
opener
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=
(?:[^*]+|\*(?!/))* =A0 # anything except */
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=
\*/=A0 =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 # comment cl=
oser
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=
([ \t]*)?=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 # trailing blanks or tabs
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 }x=20
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 { $Code .=3D " ";=
$Comments .=3D $item[1] }
=A0
I want to use the same metodology but rather than seperating the comments f=
rom the C code=A0I want to=A0use a grammar to identify such constructs and =
if any such construct is found covert them into the required output.
=A0
So please could someone help me=A0with the grammar than can be used to iden=
tify the constructs and the way I=A0can convert it into=A0a single=A0line.
=A0
Thanks=A0in advance,
=A0
Regards
Rahul Jain
HCL Technologies Ltd.=A0=A0=0A=0A=0A
--0-1332330031-1255090343=:10862
Content-Type: text/html; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable
<table cellspacing=3D"0" cellpadding=3D"0" border=3D"0" ><tr><td valign=3D"=
top" style=3D"font: inherit;"><DIV id=3Dyiv1339633584>
<DIV id=3Dyiv1637671604>Hi All,</DIV>
<DIV> </DIV>
<DIV>I am working on a C code parser, in which one of my requirements is to=
parse the C source and header files and calculate the Lines of Code. Thoug=
h there are tools to do so, all of them have a problem wherein they treat f=
unction definitions and declarations with argument list on multiple li=
nes as multiple lines rather than single line . For example </DIV>
<DIV> </DIV>
<DIV>- 1-</DIV>
<DIV> </DIV>
<DIV>/* Definition */</DIV>
<DIV>int Test ( int x,</DIV>
<DIV> &nbs=
p; int y</DIV>
<DIV> )</DIV>
<DIV>{</DIV>
<DIV>}</DIV>
<DIV> </DIV>
<DIV>or</DIV>
<DIV> </DIV>
<DIV>/* Declaration */ </DIV>
<DIV>int Test( int x,</DIV>
<DIV> &nbs=
p; int y</DIV>
<DIV> );</DIV>
<DIV> </DIV>
<DIV>is treated as 5 and 3 lines respectively rather than 3 and 1 line=
, i.e it should be treated as</DIV>
<DIV>
<DIV> </DIV>
<DIV>
<DIV>- 2 - </DIV> </DIV>
<DIV>/* Definition */</DIV>
<DIV>int Test ( int x, int y)</DIV>
<DIV>{</DIV>
<DIV>}</DIV></DIV>
<DIV> </DIV>
<DIV>
<DIV>/* Declaration */ </DIV>
<DIV>int Test( int x, int y );</DIV></DIV>
<DIV> </DIV>
<DIV>To fix this I plan to make some modifications in my perl tool. I =
wish to use RecDescent to parse the input file, identify such constucts&nbs=
p;and then use perl script to convert these multiple line construct in=
to single line contructs, so if construct - 1 - is given as input=
to the script then the output should be - 2 -. I found a script by <F=
ONT size=3D2>Damian Conway, Helmut Jarausch and <FONT size=3D2>Teodor Zlata=
nov which uses the RecDescent to seperate comments from the c code.(also at=
tached with the mail) The grammar used is </FONT></FONT></DIV>
<DIV> </DIV>
<DIV><FONT size=3D2>
<DIV>C_code : m{( </DIV>
<DIV> &nbs=
p; [^"/]+ =
# one or more non-delimiters</DIV>
<DIV> &nbs=
p; (  =
; # then (optionally)...</D=
IV>
<DIV> &nbs=
p; / =
# a potential commen=
t delimiter</DIV>
<DIV> &nbs=
p; [^*/] &=
nbsp; # which is not an actual delimiter</DIV>
<DIV> &nbs=
p; )?  =
; # </DIV>
<DIV> &nbs=
p; )+ &nbs=
p; # all repeated once or more</DIV>
<DIV> &nbs=
p; }x</DIV>
<DIV> &nbs=
p; { $Code .=3D $item[1] }</DIV>
<DIV> </DIV><FONT size=3D2>
<DIV>comment : m{ \s* =
# optional whitespace</DIV>
<DIV> &nbs=
p; // &nbs=
p; =
# comment delimiter</DIV>
<DIV> &nbs=
p; [^\n]* =
# anything except a newli=
ne</DIV>
<DIV> &nbs=
p; \n &nbs=
p; =
# then a newline</DIV>
<DIV> &nbs=
p; }x</DIV>
<DIV> &nbs=
p; { $Code .=3D "\n"; $Comm=
ents .=3D $item[1] }</DIV>
<DIV> </DIV>
<DIV> &nbs=
p; | m{\s*  =
; &=
nbsp; # optional whitespace</DIV>
<DIV> &nbs=
p; &=
nbsp; /\* =
# comment opener</DI=
V>
<DIV> &nbs=
p; &=
nbsp; (?:[^*]+|\*(?!/))* # anything except */</DIV>
<DIV> &nbs=
p; &=
nbsp; \*/  =
; # comment closer</DIV>
<DIV> &nbs=
p; &=
nbsp; ([ \t]*)? =
# trailing blanks or tabs</DIV>
<DIV> &nbs=
p; }x </DIV>
<DIV> &nbs=
p; { $Code .=3D " "; $Comments .=
=3D $item[1] }</DIV></FONT></FONT></DIV>
<DIV> </DIV>
<DIV>I want to use the same metodology but rather than seperating the comme=
nts from the C code I want to use a grammar to identify such cons=
tructs and if any such construct is found covert them into the required out=
put.</DIV>
<DIV> </DIV>
<DIV>So please could someone help me with the grammar than can be used=
to identify the constructs and the way I can convert it into a s=
ingle line.</DIV>
<DIV> </DIV>
<DIV>Thanks in advance,</DIV>
<DIV> </DIV>
<DIV>Regards</DIV>
<DIV>Rahul Jain</DIV>
<DIV>HCL Technologies Ltd. </DIV></DIV></td></tr></table><br>=0A=
=0A=0A=0A
--0-1332330031-1255090343=:10862--
--0-2040507781-1255090343=:10862
Content-Type: text/plain; name="stat-comments-dos.pl"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="stat-comments-dos.pl"
IyEgL3Vzci9iaW4vcGVybCAtdw0KIyBzdGF0LWNvbW1lbnRzLnBsIGJ5IFRl
b2RvciBabGF0YW5vdiwgdHp6QGlnbG91LmNvbQ0KIyBNYXJjaCAyNiwgMjAw
MA0KDQojIEEgc2NyaXB0IHRvIGV2YWx1YXRlIHRoZSByZWFkYWJpbGl0eSBv
ZiBjb21tZW50cw0KIyBlbWJlZGRlZCBpbiBDKysuICBVdGlsaXplcyBjb2Rl
IGZyb20gZGVtby1kZWNvbW1lbnQucGwsDQojIHdoaWNoIGlzIGluY2x1ZGVk
IHdpdGggdGhlIFBhcnNlOjpSZWNEZXNjZW50IG1vZHVsZS4NCiMgVXNlcyB0
aGUgTGluZ3VhOjpFTjo6RmF0aG9tIG1vZHVsZSB0byBldmFsdWF0ZSB0ZXh0
DQojIHJlYWRhYmlsaXR5Lg0KDQojIE9SSUdJTkFMIEJZIEhlbG11dCBKYXJh
dXNjaCANCiMgRVhURU5ERUQgQlkgRGFtaWFuIENvbndheSBBTkQgSGVsbXV0
IEphcmF1c2NoDQojIFBPTElTSEVEIEJZIFRlb2RvciBabGF0YW5vdg0KDQoN
CnVzZSBzdHJpY3Q7DQp1c2UgUGFyc2U6OlJlY0Rlc2NlbnQ7DQp1c2UgTGlu
Z3VhOjpFTjo6RmF0aG9tOw0KDQp1c2UgdmFycyBxdy8gJEdyYW1tYXIgLzsN
Cg0KbXkgJHBhcnNlciA9IG5ldyBQYXJzZTo6UmVjRGVzY2VudCAkR3JhbW1h
ciAgb3IgIGRpZSAiaW52YWxpZCBncmFtbWFyIjsNCg0KdW5kZWYgJC87DQpt
eSAkdGV4dCA9IEBBUkdWID8gPD4gOiA8REFUQT47DQoNCm15ICRwYXJ0cyA9
ICRwYXJzZXItPnByb2dyYW0oJHRleHQpIG9yIGRpZSAibWFsZm9ybWVkIEMg
cHJvZ3JhbSI7DQoNCiMgb25seSB3b3JrIHdpdGggY29tbWVudHMgb2YgbGVu
Z3RoID4gMA0KZGllICJObyBjb21tZW50cyBmb3VuZCBpbiBpbnB1dCIgdW5s
ZXNzIGxlbmd0aCAkcGFydHMtPntjb21tZW50c307DQoNCiMgY29udmVydCBl
dmVyeSBjb21tZW50IG1hcmsgdG8gYSBwZXJpb2QsIHNvIHNlcGFyYXRlIGNv
bW1lbnRzIGFyZQ0KIyBzZXBhcmF0ZSBzZW50ZW5jZXMsIGlmIHdlbGwtZm9y
bWVkLiAgTGluZ3VhOjpFTjo6RmF0aG9tIGlzIHF1aXRlDQojIGdvb2QgYXQg
ZmlndXJpbmcgb3V0IHdoYXQgc2VudGVuY2VzIGFyZSB2YWxpZCwgc28gYW4g
ZXh0cmEgcGVyaW9kDQojIGluIHRoZSB0ZXh0IHdvbid0IGFmZmVjdCB0aGUg
b3ZlcmFsbCBjb3VudHMuDQoNCiRwYXJ0cy0+e2NvbW1lbnRzfSA9fiBzIy8v
Iy4gI2c7DQokcGFydHMtPntjb21tZW50c30gPX4gcyMvXCojLiAjZzsNCiRw
YXJ0cy0+e2NvbW1lbnRzfSA9fiBzI1wqLyMuICNnOw0KDQojIHdlIGNhbiBu
b3cgZXZhbHVhdGUgdGhlIGNvbW1lbnRzIChzdG9yZWQgaW4gJHBhcnRzLT57
Y29tbWVudHN9KQ0KbXkgJGZhdGhvbSA9IG5ldyBMaW5ndWE6OkVOOjpGYXRo
b207IA0KJGZhdGhvbS0+YW5hbHlzZV9ibG9jaygkcGFydHMtPntjb21tZW50
c30pOw0KDQojIHZvaWxhLCB0aGUgcmVhZGFiaWxpdHkgcmVwb3J0IQ0KcHJp
bnQoJGZhdGhvbS0+cmVwb3J0KTsNCiAgDQpCRUdJTg0KeyAkR3JhbW1hcj08
PCdFT0YnOw0KDQpwcm9ncmFtIDogPHJ1bGV2YXI6IGxvY2FsICRXaXRoaW5D
b21tZW50PTA+DQpwcm9ncmFtIDogPHJ1bGV2YXI6IGxvY2FsICRDb21tZW50
cyA9ICIiPiAvdGhpcyBzaG91bGRuJ3QgYmUgaGVyZSA6LS8NCnByb2dyYW0g
OiA8cmVqZWN0Pg0KcHJvZ3JhbSA6IDxyZWplY3Q+IC93aXRoIHByZWp1ZGlj
ZS8NCnByb2dyYW0gOiA8cnVsZXZhcjogbG9jYWwgJENvZGUgPSAiIj4NCnBy
b2dyYW0gOiA8cnVsZXZhcjogbG9jYWwgQFN0cmluZ3M+DQoNCnByb2dyYW0J
OiA8c2tpcDonJz4gcGFydChzKQ0KCQl7IHsgY29kZT0+JENvZGUsIGNvbW1l
bnRzPT4kQ29tbWVudHMsIHN0cmluZ3M9PltAU3RyaW5nc119IH0NCg0KcGFy
dAk6IGNvbW1lbnQNCiAgICAgICAgfCBDX2NvZGUNCiAgICAgICAgfCBzdHJp
bmcNCg0KQ19jb2RlICA6IG17KAkJCQ0KCSAgICAgIFteIi9dKwkJIyBvbmUg
b3IgbW9yZSBub24tZGVsaW1pdGVycw0KCSAgICAgICgJCQkjIHRoZW4gKG9w
dGlvbmFsbHkpLi4uDQoJICAgICAgIC8JCSMgYSBwb3RlbnRpYWwgY29tbWVu
dCBkZWxpbWl0ZXINCgkgICAgICAgW14qL10JCSMgd2hpY2ggaXMgbm90IGFu
IGFjdHVhbCBkZWxpbWl0ZXINCgkgICAgICApPwkJIyANCgkgICAgKSsJCQkj
IGFsbCByZXBlYXRlZCBvbmNlIG9yIG1vcmUNCgkgICB9eA0KCQl7ICRDb2Rl
IC49ICRpdGVtWzFdIH0NCg0Kc3RyaW5nCTogbXsiCQkJIyBhIGxlYWRpbmcg
ZGVsaW1pdGVyDQoJICAgICgoCQkJIyB6ZXJvIG9yIG1vcmUuLi4NCgkgICAg
ICBcXC4JCSMgZXNjYXBlZCBhbnl0aGluZw0KCSAgICAgIHwJCQkjIG9yDQoJ
ICAgICAgW14iXQkJIyBhbnl0aGluZyBidXQgYSBkZWxpbWl0ZXINCgkgICAg
ICkqDQoJICAgICkNCgkgICAgIn14DQoJCXsgJENvZGUgLj0gJGl0ZW1bMV07
IHB1c2ggQFN0cmluZ3MsICQxIH0NCg0KDQpjb21tZW50CTogbXtccyoJCQkj
IG9wdGlvbmFsIHdoaXRlc3BhY2UNCgkgICAgLy8JCQkjIGNvbW1lbnQgZGVs
aW1pdGVyDQoJICAgIFteXG5dKgkJIyBhbnl0aGluZyBleGNlcHQgYSBuZXds
aW5lDQoJICAgIFxuCQkJIyB0aGVuIGEgbmV3bGluZQ0KCSAgIH14DQoJCXsg
JENvZGUgLj0gIlxuIjsgJENvbW1lbnRzIC49ICRpdGVtWzFdIH0NCg0KCXwg
bXtccyoJCQkjIG9wdGlvbmFsIHdoaXRlc3BhY2UNCgkgICAgL1wqCQkJIyBj
b21tZW50IG9wZW5lcg0KCSAgICAoPzpbXipdK3xcKig/IS8pKSoJIyBhbnl0
aGluZyBleGNlcHQgKi8NCgkgICAgXCovCQkgICAgICAgICMgY29tbWVudCBj
bG9zZXINCiAgICAgICAgICAgIChbIFx0XSopPyAgICAgICAgICAgIyB0cmFp
bGluZyBibGFua3Mgb3IgdGFicw0KCSAgIH14CQ0KCQl7ICRDb2RlIC49ICIg
IjsgJENvbW1lbnRzIC49ICRpdGVtWzFdIH0NCg0KRU9GDQp9DQpfX0RBVEFf
Xw0KcHJvZ3JhbSB0ZXN0OyAvLyBmb3IgZGVjb21tZW50DQoNCi8vIHVzaW5n
IFBhcnNlOjpSZWNEZXNjZW50DQoNCi8qDQogV2Ugc2hvdWxkIHJhaXNlIHRo
ZSBpbmRpY2VzIHF1aXRlIGEgYml0IHdpdGggdGhpcyB0ZXh0IHNlY3Rpb24s
DQogYmVjYXVzZSBpdCB3aWxsIGFjdHVhbGx5IGluY2x1ZGUgc2VudGVuY2Vz
IGFuZCBzdHJ1Y3R1cmUuICBTZWUsDQogdGhlIHByb2JsZW0gd2l0aCBtb3N0
IEMvQysrIHByb2dyYW1zIGlzIHRoYXQgdGhleSB1c2UgY29tbWVudHMNCiB0
aGF0IGFyZSB2ZXJ5IHNob3J0IGFuZCBjb252ZXkgbGl0dGxlIGluZm9ybWF0
aW9uLg0KKi8NCiANCmludCBtYWluKCkNCnsNCi8qIHRoaXMgc2hvdWxkDQog
ICBiZSByZW1vdmVkDQoqLw0KICBjaGFyICpjcDEgPSAiIjsNCiAgY2hhciAq
Y3AyID0gImNwMiI7DQogIGludCBpOyAgLy8gYSBjb3VudGVyDQogICAgICAg
ICAgLy8gcmVtb3ZlIHRoaXMgbGluZSBhbHRvZ2VodGVyDQogIGludCBrOyAg
DQogICAgICBpbnQgbW9yZV9pbmRlbnRlZDsgIC8vIGtlZXAgaW5kZW50YXRp
b24NCiAgICAgIGludCBsOyAgLyogYSBsb29wDQogICAgICAgICAgICAgdmFy
aWFibGUgKi8NCiAgICAgIC8vIHNob3VsZCBiZSBjb21wbGV0ZWx5IHJlbW92
ZWQNCg0KICBjaGFyICpzdHIgPSAiLyogY2VjaSBuJ2VzdCBwYXMgdW4gY29t
bWVudGFpcmUgKi8iOw0KICByZXR1cm4gMDsNCn0NCg==
--0-2040507781-1255090343=:10862--