Memory leak when rolling back from IntegrityErrors?

Joe <[email protected]>
Newsgroups gmane.comp.python.db.psycopg.devel
Message-ID <[email protected]>
I'm encountering what appears to be a memory leak when rolling back from 
IntegrityErrors. Attached is a small test script that reproduces the 
problem. Use 'top' or similar to monitor the script's memory usage. It 
should increase very quickly when run with the 'leak' argument.

I tested on two platforms:
1) Debian Etch with python 2.4.4., psycopg2 version '2.0.7 (dec mx dt 
ext pq3)' and postgresql 8.1.
2) Ubuntu 8.04 Hardy Heron with python 2.5.2, psycopg2 versions '2.0.6 
(dec mx dt ext pq3)' and postgresql 8.3.

Sincerely,
Joe

_______________________________________________
Psycopg mailing list
Psycopg-IAPFreCvJWPBWskQ1e/[email protected]
http://lists.initd.org/mailman/listinfo/psycopg
leak.test.py (text/plain, 2.3 KB)
"""
script: test_leak.py

This script attempts to repeatedly insert the same list of rows into
the database table, causing a duplicate key error to occur. It will
then roll back the transaction and try again.

Database table schema:
    -- CREATE TABLE t (foo TEXT PRIMARY KEY);

There are two ways to run the script, which will launch one of the
two functions:

# leak() will cause increasingly more RAM to be used by the script.
$ python <script_nam> leak

# noleak() does not have the RAM usage problem. The only difference 
# between it and leak() is that 'rows' is created once, before the loop.
$ python <script_name> noleak

Use Control-C to quit the script.
"""
import sys
import psycopg2

DB_NAME = '<database_name>'
DB_USER = '<database_user>'
DB_HOST = 'localhost'
DB_PASSWORD = '<database_password>'

connection = psycopg2.connect(host=DB_HOST, user=DB_USER,
                password=DB_PASSWORD, database=DB_NAME)
cursor = connection.cursor()
# Uncomment the following if table 't' does not exist
#create_table = """CREATE TABLE t (foo TEXT PRIMARY KEY)"""
#cursor.execute(create_table)

insert = """INSERT INTO t VALUES (%(foo)s)"""

def leak():
    """rows created in each loop run"""
    count = 0
    while 1:
        try:
            rows = []
            for i in range(1, 100):
                row = {'foo': i}
                rows.append(row)
            count += 1
            print "loop count:", count
            cursor.executemany(insert, rows)
            connection.commit()
        except psycopg2.IntegrityError:
            connection.rollback()

def noleak():
    """rows created once, before the loop"""
    rows = []
    for i in range(1, 100):
        row = {'foo': i}
        rows.append(row)
    count = 0
    while 1:
        try:
            count += 1
            print "loop count:", count
            cursor.executemany(insert, rows)
            connection.commit()
        except psycopg2.IntegrityError:
            connection.rollback()

usage = "%s requires one argument: 'leak' or 'noleak'" % sys.argv[0]
try:
    if 'leak' == sys.argv[1]:
        run_function = leak
    elif 'noleak' == sys.argv[1]:
        run_function = noleak
    else:
        print usage
        sys.exit()
except IndexError:
    print usage
    sys.exit()

# Run leak() or noleak(), whichever was indicated on the command line
run_function()
lmpx.com only provides a reader for public news (NNTP) servers. It is not affiliated with the servers or forums shown here and is not responsible for the content of articles, which is written by their respective authors.