Re: Race condition in list_consume()

On 16.04.2007, at 19:08, Raul Igrisan wrote:

> “and gets locked up because the list's atomic lock is locked by B.”
>
> The lock is not held by B since the wait releases the lock (and  
> reacquires it before returning).
>
> Do you have a logical explanation for the unlock/wait/lock approach?


actually, I was not able to reproduce this issue isolated.
the test application always works if the unlock/lock I added is there  
or not. *sight*.

looking at my first debug information again, I can see that several  
threads are sitting in

debug()
	gwlist_lock(writers)
		mutex_lock_real()
			semaphore_wait_signal_trap()

and never continue to run.

the only gwlist_lock() in debug is the one which locks the "writers"  
list
this is not the atomic lock but the list lock. So my "fix" is anyway  
not relevant and probably not fixing what it is supposed to fix.

For some reason we never get woken up. I suspect we might run out of  
some system resources instead so the semaphore was not created or we  
can never get called otherwise. I'm getting in touch with Apple about  
this. Semaphores are different under MacOS X and I stumbled over some  
people say they can give issues (MySQL barks about them not being so  
performant).

While reading the code I have not spotted any possible code path  
where the gwlist_lock(writers) could not end up with a pretty instant  
gwlist_unlock(writers). There is no way to not return from debug  
without gwlist_unlock(writers). There is no thread still sitting in  
debug() doing weird things (like a busy loop looking for the end of  
the string which is 2GB further down or so).



>
>
>
>
>
>
>
>
> I’ve seen this pattern in kannel sources: lock/set mutex owner to  
> -1/wait/set mutex owner to current thread/unlock
>
> list->single_operation_lock->owner = -1;
>
>
>
> Does anyone know its story?
>
>
>
>
>
>
>
> From: Andreas Fink [mailto:[email protected]]
> Sent: 16 April 2007 16:44
> To: devel Devel
> Subject: Race condition in list_consume()
>
>
>
> We found a severe bug in gwlib.
>
>
>
> We have the following scenario:
>
>
>
> A calls debug("xxx",0,"xxxx") which does :
>
>
>
> gwlist_add_producer(writers);
>
>
>
>         and continues but doesnt reach yet this line:
>
>
>
> gwlist_remove_producer(writers);
>
>
>
>
>
> at this point the list "writers" is empty but has writers- 
> >num_producers=1
>
>
>
> B does:
>
>             lock(list); /* atomic lock */
>
>
>
> list->single_operation_lock->owner = -1;
>
> pthread_cond_wait(&list->nonempty, &list->single_operation_lock- 
> >mutex);
>
>
>
> so it waits until A is calling gwlist_remove_producer()
>
>
>
> and wait until A completes.
>
>
>
>
>
> Now A is calling this:
>
>
>
> void gwlist_remove_producer(List *list)
>
> {
>
> lock(list);
>
> gw_assert(list->num_producers > 0);
>
> --list->num_producers;
>
> pthread_cond_broadcast(&list->nonempty);
>
> unlock(list);
>
> }
>
>
>
> and gets locked up because the list's atomic lock is locked by B.
>
>
>
>
>
> C now has a new debug message and gets stopped at gwlist_produce().
>
>
>
>
>
> In other words, every process who wants to write to debug log gets  
> stuck.
>
>
>
> Now there is different solutions to this.
>
> Our approach would be to do in gwlist_consume() to do this:
>
>
>
>
>
>                         unlock(list);
>
> pthread_cond_wait(&list->nonempty, &list->single_operation_lock- 
> >mutex);
>
> lock(list);
>
>
>
>
>
> Any other ideas?
>
> maybe no atomic lock around gwlist_remove_producer() ?
>
>
>
>
>
> Andreas Fink
>
>
>
> Fink Consulting GmbH
>
> Global Networks Schweiz AG
>
> BebbiCell AG
>
>
>
> ---------------------------------------------------------------
>
> Tel: +41-61-6666330 Fax: +41-61-6666331 Mobile: +41-79-2457333
>
> Address: Clarastrasse 3, 4058 Basel, Switzerland
>
> E-Mail: [email protected]
>
> www.finkconsulting.com www.global-networks.ch www.bebbicell.ch
>
> ---------------------------------------------------------------
>
> ICQ: 8239353 MSN: [email protected] AIM: smsrelay Skype: andreasfink
>
> Yahoo: finkconsulting SMS: +41792457333
>
>
>
>
>
>
>
>
>
>
>
>
>



Andreas Fink

Fink Consulting GmbH
Global Networks Schweiz AG
BebbiCell AG

---------------------------------------------------------------
Tel: +41-61-6666330 Fax: +41-61-6666331  Mobile: +41-79-2457333
Address: Clarastrasse 3, 4058 Basel, Switzerland
E-Mail:  [email protected]
www.finkconsulting.com www.global-networks.ch www.bebbicell.ch
---------------------------------------------------------------
ICQ: 8239353 MSN: [email protected] AIM: smsrelay Skype: andreasfink
Yahoo: finkconsulting SMS: +41792457333