[pysqlite] About one in a thousand inserts fail!
AK <[email protected]>
| Newsgroups | gmane.comp.python.db.pysqlite.user |
|---|---|
| Message-ID | <[email protected]> |
Hello, I'm using pysqlite for the program listed in my sig. I was
testing for possible performance issues when the number of records is
very large (I wanted to test 100k at least), and I ran into this
problem: sometimes inserts would not actually insert the record in the
db. No error would be given and program continues normally.
Now, there's probably a better way to do this, but I needed to get the
last inserted id and it seems sqlite does not have last_id command, so I
would insert a record with automatic incremented id and then query it by
using 'title' = title and 'tags' = tags, combination of which should
usually be unique (it is unique in the program, but it automatic record
inserting script it may not always be), and get the id matching these
title and tags combination. But, sometimes, there would be no results
returned! I would check the db and indeed that record would not be
there. Sometimes this would happen after a few dozen records, at other
times after 4000 records, on average after about a thousand of them. I
tried looking at the return code of insertion command, it was empty just
like with successful insertions. I changed to using the insertion in
form of giving '?', '?', [item1, item2], forget what this is called but
it didn't help.
So, here's some code:
t = 'INSERT INTO items VALUES (NULL, ?, ?, ?, CURRENT_DATE, \
CURRENT_DATE, ?, CURRENT_TIMESTAMP)'
ret = self.cursor.execute(t, [title, tags, "", filename])
cmd = 'SELECT id FROM items WHERE title="%s" AND tags="%s"' %
(title, tags)
self.cursor.execute(cmd)
fetch = self.cursor.fetchmany()
try:
lastid = fetch[0][0]
except TypeError, IndexError:
for i in range(100):
time.sleep(0.05)
cmd = 'SELECT id FROM items WHERE title="%s" AND
tags="%s"' % (title, tags)
self.cursor.execute(cmd)
fetch = self.cursor.fetchone()
if fetch:
lastid = fetch[0]
break
if not fetch:
sys.exit()
'items' table is this:
CREATE TABLE items (
id INTEGER PRIMARY KEY,
title VARCHAR(40),
tags VARCHAR(900),
content MEDIUMBLOB,
created DATE,
modified DATE,
filename VARCHAR(500),
modified_ts TIMESTAMP)
example last output in log right before the error is (first two are
title and tags before inserting them):
'eventually'
'content'
'in def insert'
'inserting item..'
'insert cmd:INSERT INTO items VALUES (NULL, ?, ?, ?,
CURRENT_DATE, CURRENT_DATE, ?, CURRENT_TIMESTAMP)'
'selecting last id..'
'cmd is,'
'SELECT id FROM items WHERE title="eventually" AND tags="content"'
'after execution of cmd'
'fetching one record..'
'fetch:'
'ret code is: '
Here is the error from console (4044 is the number of item inserted
during this run):
#4044,
Traceback (most recent call last):
File "test10k.py", line 55, in ?
main()
File "test10k.py", line 45, in main
dbid = db.insert([title, tag_string, body, ""]) # tags as string!
File "f:\0home\projects\Tobu\tobu\db.py", line 274, in insert
lastid = fetch[0][0]
IndexError: list index out of range
fetch[0][0] means first item out of first tuple of results.
When insertion works, it works perfectly, i.e. things end up in the
right places, title, tags, text..
I don't know what else I can try here. I'm using latest release of
pysqlite on windows, compiled
binary for python 2.4
thanks!
--
-ak
Tobu | http://www.lightbird.net/tobu/ | Freeform DB / Tagger / PIM