Re: Further adventures in the land of 64-bit linux

On Mon, Sep 16, 2002 at 04:01:31PM +0200, Martin v. Löwis wrote:
> Anders Qvist <[email protected]> writes:
> 
> > > I'm guessing that it's possible that the gr_passwd slot can be NULL on
> > > that platform; the code doesn't expect this.
> 
> Can you please try the attached patch?
> RCS file: /cvsroot/python/python/dist/src/Modules/grpmodule.c,v
> retrieving revision 2.19
> diff -u -r2.19 grpmodule.c
> --- grpmodule.c	2 Aug 2002 02:27:13 -0000	2.19
> +++ grpmodule.c	16 Sep 2002 13:58:56 -0000
> @@ -58,7 +58,12 @@
[snip]

Like a charm.

Next one to "fail" is test_longexp, which bloats the interpreter
massively. Simplified, it looks like this:

>>> l = eval ("[" + "2," * 65580 + "]")

./python ../python/dist/src/Lib/test/test_longexp.py & ; pid=$\! ;
while /bin/true ; do ps aux | grep $pid ; sleep 1 ; done

USER      PID %CPU %MEM   VSZ   RSS   TTY   STAT START   TIME COMMAND
quest   30018  0.0  0.3  3792   224 pts/5   DN   22:14   0:00 ./python ../pytho
quest   30018 10.3  4.1 10224  2536 pts/5   DN   22:14   0:00 ./python ../pytho
quest   30018 43.8 15.7 17032  9640 pts/5   RN   22:14   0:00 ./python ../pytho
quest   30018 63.7 35.1 29392 21488 pts/5   RN   22:14   0:01 ./python ../pytho
quest   30018 59.2 55.4 42000 33912 pts/5   RN   22:14   0:02 ./python ../pytho
quest   30018 32.5 65.9 64840 40376 pts/5   DN   22:14   0:04 ./python ../pytho

The behaviour seems exponential, for nothing untoward is apparent with
small number of elements in the list. As numbers grow into tens of
thousands, the memory usage starts to balloon. Technically, the
machine could prolly pass the test, had it not been for its shortage
of memory.

[ 22:40 ] - ./python 
Python 2.3a0 (#2, Sep 16 2002, 17:37:08) 
[GCC egcs-2.91.66 19990314/Linux (egcs-1.1.2 release)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> l = eval ("[" + "2," * 30000 + "]")
[47434 refs]

  PID USER     PRI  NI  SIZE  RSS SHARE STAT  LIB %CPU %MEM   TIME COMMAND
30153 quest      5   0 24952  24M  2464 S     23M  0.0 20.3   0:04 python

It's late, so I'll have to continue the hunt later. Here's the
backtrace from gdb, but it doesn't seem to point to what's wrong.

#0  0x12011e3ec in PyNode_AddChild (n1=0x200026166c8, type=301, str=0x0, 
    lineno=1) at ../python/dist/src/Parser/node.c:95
#1  0x12011ea9c in push (s=0x1202eb798, type=301, d=0x12026daa0, newstate=1, 
    lineno=1) at ../python/dist/src/Parser/parser.c:126
#2  0x12011eeb4 in PyParser_AddToken (ps=0x1202eb798, type=2, 
    str=0x2000260cc98 "2", lineno=1, expected_ret=0x11ffff2cc)
    at ../python/dist/src/Parser/parser.c:234
#3  0x120010e08 in parsetok (tok=0x1202e9b98, g=0x12026e788, start=258, 
    err_ret=0x11ffff2a8, flags=0) at ../python/dist/src/Parser/parsetok.c:157
#4  0x120010a10 in PyParser_ParseStringFlagsFilename (
    s=0x200008b604c "[2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2"..., 
    filename=0x0, g=0x12026e788, start=258, err_ret=0x11ffff2a8, flags=0)
    at ../python/dist/src/Parser/parsetok.c:56
#5  0x1200108d8 in PyParser_ParseStringFlags (
    s=0x200008b604c "[2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2"..., 
    g=0x12026e788, start=258, err_ret=0x11ffff2a8, flags=0)
    at ../python/dist/src/Parser/parsetok.c:31
#6  0x1200f7e94 in PyParser_SimpleParseStringFlags (
    str=0x200008b604c "[2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2"..., 
    start=258, flags=0) at ../python/dist/src/Python/pythonrun.c:1183
#7  0x1200f7508 in PyRun_StringFlags (
    str=0x200008b604c "[2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2"..., 
    start=258, globals=0x120285780, locals=0x120285780, flags=0x11ffff360)
    at ../python/dist/src/Python/pythonrun.c:1040
#8  0x1200a3c60 in builtin_eval (self=0x0, args=0x20000048998)
    at ../python/dist/src/Python/bltinmodule.c:467
---Type <return> to continue, or q <return> to quit---
#9  0x12012ef5c in PyCFunction_Call (func=0x2000002ef18, arg=0x20000048998, 
    kw=0x0) at ../python/dist/src/Objects/methodobject.c:80
#10 0x1200c0ab8 in call_function (pp_stack=0x11ffff468, oparg=1)
    at ../python/dist/src/Python/ceval.c:3228
#11 0x1200bab40 in eval_frame (f=0x1202908a0)
    at ../python/dist/src/Python/ceval.c:1993
#12 0x1200bda30 in PyEval_EvalCodeEx (co=0x20000026818, globals=0x120285780, 
    locals=0x120285780, args=0x0, argcount=0, kws=0x0, kwcount=0, defs=0x0, 
    defcount=0, closure=0x0) at ../python/dist/src/Python/ceval.c:2538
#13 0x1200afdf4 in PyEval_EvalCode (co=0x20000026818, globals=0x120285780, 
    locals=0x120285780) at ../python/dist/src/Python/ceval.c:478
#14 0x1200f77c8 in run_node (n=0x2000003f038, 
    filename=0x11ffffb6f "../python/dist/src/Lib/test/test_longexp.py", 
    globals=0x120285780, locals=0x120285780, flags=0x11ffff898)
    at ../python/dist/src/Python/pythonrun.c:1083
#15 0x1200f7714 in run_err_node (n=0x2000003f038, 
    filename=0x11ffffb6f "../python/dist/src/Lib/test/test_longexp.py", 
    globals=0x120285780, locals=0x120285780, flags=0x11ffff898)
    at ../python/dist/src/Python/pythonrun.c:1070
#16 0x1200f7698 in PyRun_FileExFlags (fp=0x12027c560, 
    filename=0x11ffffb6f "../python/dist/src/Lib/test/test_longexp.py", 
    start=257, globals=0x120285780, locals=0x120285780, closeit=1, 
    flags=0x11ffff898) at ../python/dist/src/Python/pythonrun.c:1061
#17 0x1200f57e0 in PyRun_SimpleFileExFlags (fp=0x12027c560, 
    filename=0x11ffffb6f "../python/dist/src/Lib/test/test_longexp.py", 
    closeit=1, flags=0x11ffff898) at ../python/dist/src/Python/pythonrun.c:692
#18 0x1200f4bc8 in PyRun_AnyFileExFlags (fp=0x12027c560, 
    filename=0x11ffffb6f "../python/dist/src/Lib/test/test_longexp.py", 
    closeit=1, flags=

Could it be these lines in parsetok.c (138-139)?

  len = b - a; /* XXX this may compute NULL - NULL */
  str = (char *) PyObject_MALLOC(len + 1);

CVS/Entries says: 

  /parsetok.c/2.33/Sun Aug  4 17:29:52 2002//
-- 
Anders "Quest" Qvist

"We've all heard that a million monkeys banging on a million typewriters
will eventually reproduce the entire works of Shakespeare. Now, thanks 
to the Internet, we know this is not true." -- Robert Wilensky