Memory leak when rolling back from IntegrityErrors?
Joe <[email protected]>
| Newsgroups | gmane.comp.python.db.psycopg.devel |
|---|---|
| Message-ID | <[email protected]> |
I'm encountering what appears to be a memory leak when rolling back from IntegrityErrors. Attached is a small test script that reproduces the problem. Use 'top' or similar to monitor the script's memory usage. It should increase very quickly when run with the 'leak' argument. I tested on two platforms: 1) Debian Etch with python 2.4.4., psycopg2 version '2.0.7 (dec mx dt ext pq3)' and postgresql 8.1. 2) Ubuntu 8.04 Hardy Heron with python 2.5.2, psycopg2 versions '2.0.6 (dec mx dt ext pq3)' and postgresql 8.3. Sincerely, Joe _______________________________________________ Psycopg mailing list Psycopg-IAPFreCvJWPBWskQ1e/[email protected] http://lists.initd.org/mailman/listinfo/psycopg
leak.test.py
(text/plain, 2.3 KB)
"""
script: test_leak.py
This script attempts to repeatedly insert the same list of rows into
the database table, causing a duplicate key error to occur. It will
then roll back the transaction and try again.
Database table schema:
-- CREATE TABLE t (foo TEXT PRIMARY KEY);
There are two ways to run the script, which will launch one of the
two functions:
# leak() will cause increasingly more RAM to be used by the script.
$ python <script_nam> leak
# noleak() does not have the RAM usage problem. The only difference
# between it and leak() is that 'rows' is created once, before the loop.
$ python <script_name> noleak
Use Control-C to quit the script.
"""
import sys
import psycopg2
DB_NAME = '<database_name>'
DB_USER = '<database_user>'
DB_HOST = 'localhost'
DB_PASSWORD = '<database_password>'
connection = psycopg2.connect(host=DB_HOST, user=DB_USER,
password=DB_PASSWORD, database=DB_NAME)
cursor = connection.cursor()
# Uncomment the following if table 't' does not exist
#create_table = """CREATE TABLE t (foo TEXT PRIMARY KEY)"""
#cursor.execute(create_table)
insert = """INSERT INTO t VALUES (%(foo)s)"""
def leak():
"""rows created in each loop run"""
count = 0
while 1:
try:
rows = []
for i in range(1, 100):
row = {'foo': i}
rows.append(row)
count += 1
print "loop count:", count
cursor.executemany(insert, rows)
connection.commit()
except psycopg2.IntegrityError:
connection.rollback()
def noleak():
"""rows created once, before the loop"""
rows = []
for i in range(1, 100):
row = {'foo': i}
rows.append(row)
count = 0
while 1:
try:
count += 1
print "loop count:", count
cursor.executemany(insert, rows)
connection.commit()
except psycopg2.IntegrityError:
connection.rollback()
usage = "%s requires one argument: 'leak' or 'noleak'" % sys.argv[0]
try:
if 'leak' == sys.argv[1]:
run_function = leak
elif 'noleak' == sys.argv[1]:
run_function = noleak
else:
print usage
sys.exit()
except IndexError:
print usage
sys.exit()
# Run leak() or noleak(), whichever was indicated on the command line
run_function()