Speeding up P::RD using a queue...

on 2001/12/11 18:55 Terrence Brannon wrote:

>** this recent discussion: http://www.perlmonks.org/index.pl?node_id=130656

Heh. Ironic to see my thread on perlmonks posted here as a link. :-) 
Nevertheless I still have yet to get a satisfactory answer to the post I 
made there http://www.perlmonks.org/index.pl?node_id=130714

Basically the idea was to speed up P::RD by making it process line by line 
from a queue.  Each time a rule succesfully fired it would pull a new line 
from the queue and append it to $text.  Which worked for my test data set 
when shuffled, but not when it was sorted.

use strict;
use warnings;
use Parse::RecDescent;
use Data::Dumper;

our $RD_TRACE=1;

my $Grammar=<<'END_GRAMMAR';

{my $company=""}

startrule   :   file

file        :   header record(s?) trailer_t
                {
                    
$return={header=>$item[1],records=>$item[2],count=>$item[3]};
                }

header      :   header_t data_t
                {
                    $return={company=>$item[1],code=>$item[2]};
                      $text.=shift @::text;
                      print "Company Set to $company\n Text=$text";
                }

record      :   valid_record | <error>

valid_record:   type_t ',' number_t ',' number_t(?) ',' "$company"
                {
                    $return=[ $item[1],$item[3],@{$item[5]} ? $item[5] : 
undef ];
                    $text.=shift @::text; #grab a line from queue
                    print "Text=$text";
                }

header_t    :   /HDR\w+/ {$return=substr($item[1],3); $company=$return;}
trailer_t   :   /TLR\d+/ {$return=substr($item[1],3)}

data_t      :   /\w+/
type_t      :   /ADD(?:RANGE)?|DELETE(?:RANGE)?/
number_t    :   /\d+/

END_GRAMMAR


my $parser = Parse::RecDescent->new($Grammar) or die "Bad grammar!\n";


our @text=<DATA>;

if (defined( my $t=$parser->startrule(shift @text))) {
    print Dumper($t);
} else {
    print "Bad text!\n";
}


__DATA__
HDRCOMPNAME BIG000OLD111IDENTIFIER1020301WITH1010LOTS1010OF1010CRAP
ADD,1234567890,,COMPNAME
ADD,1234567891,,COMPNAME
ADD,1234567892,,COMPNAME
ADDRANGE,1468,1680,COMPNAME
ADDRANGE,2468,2680,COMPNAME
ADDRANGE,3468,3680,COMPNAME
DELETE,987654321,,COMPNAME
DELETE,987654322,,COMPNAME
DELETE,987654323,,COMPNAME
DELETERANGE,13579,13599,COMPNAME
DELETERANGE,23579,23599,COMPNAME
DELETERANGE,33579,33599,COMPNAME
TLR000012

Now weirdly it seems to perform as expected if the same data is organized as 
so:

HDRCOMPNAME BIG000OLD111IDENTIFIER1020301WITH1010LOTS1010OF1010CRAP
ADD,1234567890,,COMPNAME
ADDRANGE,1468,1680,COMPNAME
DELETE,987654321,,COMPNAME
DELETERANGE,13579,13599,COMPNAME
ADD,1234567891,,COMPNAME
ADDRANGE,2468,2680,COMPNAME
DELETE,987654322,,COMPNAME
DELETERANGE,33579,33599,COMPNAME
ADD,1234567892,,COMPNAME
ADDRANGE,3468,3680,COMPNAME
DELETE,987654323,,COMPNAME
DELETERANGE,23579,23599,COMPNAME
TLR000012

A trace of the the two is identical until the second record is encountered, 
where the first data simply reports that it was able to find only 1 record, 
and the second parses the whole lot.

I was able to overcome this by changing valid_rule to

valid_record:   {
			while ($lines<4) {
				$text.=shift(@::text);
				$lines++;
			}
                	warn "Text=$text";
                }
		type_t ',' number_t ',' number_t(?) ',' "$company"
                {
                	$return=[
                                 $item[2],
                                 $item[4],
                                 @{$item[6]} ? $item[6]->[0] : undef
                                ];
                	$lines--;
                }

And adding a my declaration for $line to the preparse action.  Note that 
this tries to keep 4 lines in $text at all times, 3 did not work!. I havent 
the foggiest why this works, nor what is going on, and accordingly I dont 
trust it in the slightest. (It is notable that there are triplets of the 
same record type, is n+1 relevent? I dont know)  It was suggested it had to 
do with changes to $text only applying if a rule is successful, but this 
does not explain why record(s?) fails after the first record when type_t is 
the same on consecutive records, but not when it is different.

Anyway, any thoughts or ideas would be welcome.

Oh and any workable ideas that I get will be forwarded onto perlmonks unless 
requested not to.

Yves


_________________________________________________________________
Send and receive Hotmail on your mobile device: http://mobile.msn.com