Re: performance improvements for 1.3

By in-line concatenation, I mean that a construction like

    Hello, <% $name %>, how are you?

gets converted to

    $$_outbuf .= 'Hello, ';
    $$_outbuf .= $name;
    $$_outbuf .= ', how are you?'

instead of the current

    $m->print('Hello, ');
    $m->print($name);
    $m->print(', how are you?');

where $m->print ultimately does the same concatenation.

Yes, this is a lot of concatenation to a single buffer. It would be more 
efficient to use join, as Benjamin suggests - Dave R tried this 
optimization once. Unfortunately this changes the order of execution of 
<% %> expressions in a non-inintuive way. For example it completely 
breaks the

    <% $m->call_next %>

idiom. For more, see http://marc.theaimsgroup.com/?t=101962664900001&r=1&w=2

 > Perl will allocate a new string to hold two of them, copy them to the
 > new memory, then repeat this for the new string and the leftover
 > string.  As you can see, as N gets high, you're doing a -lot- of
 > memory and copy work.

Are you sure? If Perl has enough extra space allocated for the string, 
it can do concatenation in place without copying the original string, right?

Perl seems to be very smart about deciding how big of a buffer to 
reserve for strings. I ran a test where I concatenated five characters 
at a time to a string up to 200K and counted the number of times the 
underlying string pointer changed. Perl only had to reallocate seven 
times. Moreover, if you repeated the test with the same lexical scalar 
(even if you exited and reentered the lexical scope), Perl seemed to 
remembered how much to allocate to that scalar. Anyway, if I can find 
the analysis I'll send it out.

As part of the optimizations I'm checking in, there's a new Interp 
parameter called buffer_preallocate_size that preallocates the output 
buffer to a particular size, to avoid even those few reallocs.

HTH. I'd be interested in hearing if anyone with deep Perl internals 
concatentation knowledge can comment, as my experiments are purely 
behavioral.


Benjamin Kram wrote:

>>By in-line concatenation, you mean "foo" . "bar"?  I seem to remember
>>reading once that that was relatively inefficient as it was constantly
>>reallocating memory to hold the result.
>>
>>I can try and find a cite if that's helpful.
> 
> 
> It isn't efficient to do 
>  "string1" . "string2" . "string3" . "stringN"
> Perl will, (in order of expression evaluation) allocate a new string to hold two of them, copy them to the new memory, then repeat this for the new string and the leftover string.  As you can see, as N gets high, you're doing a -lot- of memory and copy work.
> 
> That's what join() is for...
> 
> cheers,
> b
> 



-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php