Re: How does GCC > 3.* do the padding in memory allocation?

Rick Zhong wrote:

> I am trying the Alpha One's "Smashing The Stack For Fun And Profit". But i
> realise that the new GCC compiler has add a random padding between the last
> local variable and return address during the memory allocation.
> 
> For example:
> void function(int a, int b, int c)
> {
>     int z[1];
> }
> 
> For for this arrray z, the gcc behave normally for z[1], z[2] , it will
> allocate 4 byte for z[1], and 8 byte for z[2].  When I increase to z[3] and
> z[4] the compiler will allocate 24 bytes.  When z[5] is 5, it will allocate
> 40 bytes.
> 
> My question is how the compile do the padding? Any specific algorithm?

AFAICT, the size of the stack frame (including the 8 bytes for the
saved EIP and EBP) is always a multiple of 16, except when the local
variables occupy exactly 1, 2, 4 or 8 bytes.

Experimentation suggests that, in the general case, the amount
explicitly subtracted from ESP (not including the pushed EIP or EBP)
is:

	((n+15)/16)*16+8

where n is the number of bytes which the local variables would
require. IOW, if n isn't a multiple of 16, round it upwards to the
next multiple of 16, then add 8.

The exception is that for n=1, n=2 and n=4, it subtracts 4 bytes from
ESP, while for n=8 it subtracts 8 bytes.

You can confirm this by generating a file comprising functions of the
form:

	void f1(void) { char d[1]; }
	void f2(void) { char d[2]; }
	void f3(void) { char d[3]; }
	...

compiling it to assembler with "gcc -S", then extract the "subl"
instructions with grep.

-- 
Glynn Clements <[email protected]>