Re: [bdbxml] XmlManager::query hangs sometimes

All,

I'm not sure this ever got back to the list.

The problem is a deadlock between BDB XML and the
Python Global Interpreter Lock (GIL).  There is an actual bug
in BDB XML where it needs to handle the GIL.

This is being addressed.  If anyone is using python and
threads, and has a hang/deadlock issue, let me know.
The fix is not yet complete.

Regards,

George

> Mihai,
>
> Let's take this one off-list.   I'll summarize the resolution when  
> we're
> done.
>
> Regards,
>
> George
>
>> Hi and thanks for helping me so far.
>>
>> I added deadlock handling, but it still hangs! :(
>>
>>> As John has pointed out, you may need DB_THREAD in your environment.
>>
>> Added.
>> When I open the environment the second time (first time is  
>> recovery, indexes,
>> etc.), tha flags are:
>> DB_INIT_LOCK, DB_INIT_LOG, DB_INIT_MPOOL, DB_INIT_TXN, DB_CREATE,  
>> DB_THREAD
>>
>>> However, if you have concurrent operations going on, and using
>>> transactions, as you are, you need to handle deadlocks.
>>> There are methods on the DBEnv object that allow you to specify
>>> deadlock detection and handling.   If you do not, you will  
>>> experience
>>> hangs in your application.  The alternative is to always serialize
>>> access to your containers.
>>
>> I chose to implement a deadlock handler. It runs in a separate  
>> thread and
>> every 100ms (I will tune it later) it calls
>> myEnvironmentInstance.lock_detect(DB_LOCK_RANDOM)
>>
>>> One other point, when creating a transactional server, you also need
>>> to be able to run recovery on your environment, which means
>>> adding the DB_RECOVER flag to your environment.  However, you
>>> must be sure that the environment creation/open is happening
>>> exclusively.  That is, no other processes are threads can be
>>> accessing the environment when it is opened for recovery.
>>
>> Did this too.
>>
>>>
>>
>> It only hangs when it does a read and a write concurently. Write 
>> +write and
>> read+read don't hang (tested with parallel looped operations, no  
>> sleep() ).
>> In my case, when it hangs the operations are:  
>> XmlContainer::putDocument(...)
>> and XmlManager::query(...) . The operations reffer to the same  
>> container. The
>> hang happens always, when the 2 calls are concurent.
>>
>> The strangest thing is that ALL threads hang, including the  
>> DeadLock handler
>> thread! I tryed running it in a python command line interpretor  
>> (as the
>> operations and deadlock handling are made in new threads, the main  
>> thread is
>> not busy, so i can use the interpreter while the server is  
>> serving), but when
>> it hangs, the interpreter hangs too! I also tryed to create a  
>> thread totaly
>> unrelated to the DB, and it hangs too.
>>
>> This makes me think of the python global lock
>> ( http://docs.python.org/api/threads.html )
>> Python uses a global, per-process lock, to make all python  
>> operations and
>> C/API calls (!) atomic between threads, but it releases the lock  
>> around
>> blocking operations. Could this be related? Let's not forget that  
>> read+read
>> and write+write do NOT hang.
>>
>> strace shows, of course that all threads are in futex(... calls.  
>> There is a
>> total of exactly 2 futex ids that show up. None of my threads  
>> explicitly use
>> locks (becuase I don't need them, as the thread-safety provided by  
>> the python
>> global lock suits my needs)
>>
>> Tests showed the problem is not related to the remote request  
>> handling logic,
>> or any other auxiliar stuff.
>>
>> I run db_stat and db_deadlock but strangely they always show the same
>> information (same number of started/commited/aborted transactions,  
>> same
>> number of locks(0), etc.). Am I not using them right? I run them with
>> -h /path/to/the/db, the DB_CONFIG file does not exist.
>>
>> Versions:
>> dbxml package (libs, bins, dbxml python module): 4.3.29:  
>> (September  6, 2005)
>> bsddb3 python module 4.3.3
>> python 2.4
>>
>> I hope I didn't bore you with all the details :)
>>
>> Thanks
>>
>> -Mihai
>
>
>
> ------------------------------------------
> To remove yourself from this list, send an
> email to [email protected]
>



------------------------------------------
To remove yourself from this list, send an
email to [email protected]