Re: Question regarding IPIQs

On 28 August 2013 23:35, Matthew Dillon <[email protected]> wrote:

> :Hi there,
> :
> :I've been studying dragonfly for a while now and this is something I
> :could never figure out, I'm pretty sure I'm missing something and I'd
> :appreciate if someone could give me some pointers.
> :
> :Suppose the following cenario.
> :
> :Cenario A [A]
> :cpu0: on whichever context
> :cpu0: spin_lock(&a) (now in a critical section)
> :cpu0: wakeup(ident1) (ident1 has a sleeper on cpu1)
> :cpu0: lwkt_send_ipiq3(cpu1, wakeup(ident1))
>
>     Illegal.  You should not call wakeup() with a spin lock held.
>     You most definitely should not be calling any IPIQ functions with
>     a spin lock held.
>
>     You have two solutions here.  First, use a regular lock or a token
>     instead of a spin lock.  Second, flag the condition and issue the
>     wakeup() or IPIQ function after releasing the spin lock.
>
>     In DragonFly, spin locks are only intended to be used for very
>     short sequences of self-contained or mostly self-contained code.
>
> :But ipiq for cpu1 is full, so I enable interrupts, although I'm still on
> :a critical section, process my own ipiq and spin until cpu1 queue is not
> :full, but at this point I'm still holding spin_lock a.
>
>     Critical sections do not disable interrupts, they simply prevent normal
>     preemptive processing (preemption by an interrupt thread or IPI)
>     from occuring.  Interrupts will still occur, but they will simply flag
>     the pending condition in mycpu->gd_reqflags, EOI the interrupt, and
>     immediately return.  The interrupt is then processed normally when the
>     critical section is exited.
>
>     However, if you issue an IPIQ operation inside a critical section which
>     would otherwise be forced to spin-wait due to a full IPIQ queue, it
>     will synchronously execute any incoming IPIs while it is spin-waiting
>     to avoid deadlocking the system.
>
> :Both realize the other is full and process their own queues, but they
> break the
> :atomicity required by the two calls to crit_enter() earlier
> :
> :What am I missing ? (Sorry for the noise).
> :
> :Thanks
>
>     Programs which enter critical sections can always assume that there
> will
>     be no asynchronous atomicy breakage, but there are no guarantees
> regarding
>     synchronous atomicy.  For example, you can tsleep() just fine while
>     holding a critical section.  Nominal IPIQ operations fall under the
> same
>     rules.  Incoming IPIs will not be executed asynchronously but they can
>     still be executed synchronously if your outgoing IPI operation would
>     otherwise be forced to spin-wait.
>
>
Thanks for the detailed reply.

I assumed synchronous atomicity had to be kept across a critical section, I
had considered the tsleep case but thought it was special, since you're
sleeping, it's already implied you accept losing atomicity.