RE: Further adventures in the land of 64-bit linux

[Anders Qvist, on test_longexp]
> ...
> The behaviour seems exponential, for nothing untoward is apparent with
> small number of elements in the list.

The parse tree built for this list is huge, though.  test_longexp is a pain
in the ass that's failed to work on various platforms since it was first
introduced.  That got fixed for all known platforms after 2.2.1 was
released, via a combination of switching to pymalloc for the small
allocations, and doing aggressive overallocation for monstrously large (many
children) parse nodes.

> As numbers grow into tens of thousands, the memory usage starts to
> balloon.

This is so on all platforms -- the test is extreme.

> Technically, the machine could prolly pass the test, had it not been
> for its shortage of memory.
>
> [ 22:40 ] - ./python
> Python 2.3a0 (#2, Sep 16 2002, 17:37:08)
> [GCC egcs-2.91.66 19990314/Linux (egcs-1.1.2 release)] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
> >>> l = eval ("[" + "2," * 30000 + "]")
> [47434 refs]
>
>   PID USER     PRI  NI  SIZE  RSS SHARE STAT  LIB %CPU %MEM   TIME COMMAND
> 30153 quest      5   0 24952  24M  2464 S     23M  0.0 20.3   0:04 python
>
> It's late, so I'll have to continue the hunt later. Here's the
> backtrace from gdb, but it doesn't seem to point to what's wrong.

To the contrary, I expect it pointed directly at the culprit <wink>:

> #0  0x12011e3ec in PyNode_AddChild (n1=0x200026166c8, type=301, str=0x0,
>     lineno=1) at ../python/dist/src/Parser/node.c:95

and line 95 is a realloc() call.  As a parse node grows large,
PyNode_AddChild doubles the amount of memory it asks for every time it runs
out of room.

[Guido]
> Could it be that the parse tree uses twice the memory on a 64-bit
> machine, and that that is simply too much?  Give it a try with smaller
> values of 65580. (:-)

test_longexp consumes about 25MB of data space on my 32-bit box.  A parse
node contains two pointers so at least those two fields are twice as big on
a 64-bit box.  It may (or may not) help to rearrange the struct decl to
order the fields from widest to narrowest:

Current:

typedef struct _node {
    short		n_type;
    char		*n_str;
    int			n_lineno;
    int			n_nchildren;
    struct _node	*n_child;
} node;

Possibly more space-efficient:

typedef struct _node {
    char		*n_str;
    struct _node	*n_child;
    int			n_lineno;
    int			n_nchildren;
    short		n_type;
} node;

At least one n_child vector grows very large in test_longexp.