[pysqlite] About one in a thousand inserts fail!

AK <[email protected]>
Newsgroups gmane.comp.python.db.pysqlite.user
Message-ID <[email protected]>
Hello, I'm using pysqlite for the program listed in my sig. I was 
testing for possible performance issues when the number of records is 
very large (I wanted to test 100k at least), and I ran into this 
problem: sometimes inserts would not actually insert the record in the 
db. No error would be given and program continues normally.

Now, there's probably a better way to do this, but I needed to get the 
last inserted id and it seems sqlite does not have last_id command, so I 
would insert a record with automatic incremented id and then query it by 
using 'title' = title and 'tags' = tags, combination of which should 
usually be unique (it is unique in the program, but it automatic record 
inserting script it may not always be), and get the id matching these 
title and tags combination. But, sometimes, there would be no results 
returned! I would check the db and indeed that record would not be 
there. Sometimes this would happen after a few dozen records, at other 
times after 4000 records, on average after about a thousand of them. I 
tried looking at the return code of insertion command, it was empty just 
like with successful insertions. I changed to using the insertion in 
form of giving '?', '?', [item1, item2], forget what this is called but 
it didn't help.

So, here's some code:

        t = 'INSERT INTO items VALUES (NULL, ?, ?, ?, CURRENT_DATE, \
                CURRENT_DATE, ?, CURRENT_TIMESTAMP)'
        ret = self.cursor.execute(t, [title, tags, "", filename])
        cmd = 'SELECT id FROM items WHERE title="%s" AND tags="%s"' % 
(title, tags)
        self.cursor.execute(cmd)
        fetch = self.cursor.fetchmany()
        try:
            lastid = fetch[0][0]
        except TypeError, IndexError:
            for i in range(100):
                time.sleep(0.05)
                cmd = 'SELECT id FROM items WHERE title="%s" AND 
tags="%s"' % (title, tags)
                self.cursor.execute(cmd)
                fetch = self.cursor.fetchone()
                if fetch:
                    lastid = fetch[0]
                    break
            if not fetch:
                sys.exit()

'items' table is this:
CREATE TABLE items (
        id INTEGER PRIMARY KEY,
        title VARCHAR(40),
        tags VARCHAR(900),
        content MEDIUMBLOB,
        created DATE,
        modified DATE,
        filename VARCHAR(500),
        modified_ts TIMESTAMP)

example last output in log right before the error is (first two are 
title and tags before inserting them):

'eventually'
'content'
'in def insert'
'inserting item..'
'insert cmd:INSERT INTO items VALUES (NULL, ?, ?, ?, 
CURRENT_DATE,                 CURRENT_DATE, ?, CURRENT_TIMESTAMP)'
'selecting last id..'
'cmd is,'
'SELECT id FROM items WHERE title="eventually" AND tags="content"'
'after execution of cmd'
'fetching one record..'
'fetch:'
'ret code is: '

Here is the error from console (4044 is the number of item inserted 
during this run):

 #4044,

Traceback (most recent call last):
  File "test10k.py", line 55, in ?
    main()
  File "test10k.py", line 45, in main
    dbid = db.insert([title, tag_string, body, ""])   # tags as string!
  File "f:\0home\projects\Tobu\tobu\db.py", line 274, in insert
    lastid = fetch[0][0]
IndexError: list index out of range

fetch[0][0] means first item out of first tuple of results.

When insertion works, it works perfectly, i.e. things end up in the 
right places, title, tags, text..

I don't know what else I can try here. I'm using latest release of 
pysqlite on windows, compiled
binary for python 2.4

thanks!

--
 -ak
  Tobu | http://www.lightbird.net/tobu/ | Freeform DB / Tagger / PIM
lmpx.com only provides a reader for public news (NNTP) servers. It is not affiliated with the servers or forums shown here and is not responsible for the content of articles, which is written by their respective authors.