Re: [Rtlinuxgpl] rtl_hard_enable_irq() and system hand/lockup

>> 1. Your rtl driver is called for interrupt 'I'
>> 2. The interrupt wasn't really for your baord, so you do nothing, but at
>>     the end of your function you rtl_hard_enable_irq(I).
>> 3. Immediately the interrupt controller is told that 'I' is enabled, which
>>     causes the rtl_intercept() interrupt handler to run again (it preempts
>>     itself!).
>> 4. The rtl_intercept() handler sees your code wants IRQ 'I', so goto (1)
>>     above.. the infinite loop is created and the system hangs.
>
> that would result in a stack overflow sooner or later - but I don't think
> this model is true for edge triggert IRQs - it is though for level triggert
> ones as the rt-handler is not resetting them.

Ok, so edge triggered IRQs is what I am using (PCI bus), so it can't be 
the case that the interrupt is always necessarily on, so it won't be an 
infinite loop necessarily... fair enough.

However, I am sharing this IRQ line with like 5 other devices.. some of 
which generate interrupts frequently (such as a network board)... could 
there then be a stack overflow created very quickly by this constant 
preemption of the rt-handler if the interrupts come in fast enough.. or is 
that extremely unlikely?  Do some boards keep generating interrupts for 
every clock cycle when their internal buffers fill, etc?

Also what happens if the IRQ is shared amongst many linux devices and for 
example my rt-handler keeps re-enabling it, and drivers' handlers end up 
pre-empting themselves?  Or does the IRQ not make its way to linux more 
than once...? (since internally linux thinks of the IRQ as off while it is 
servicing it..).

Hmm.. I should read the sources more, I think.. :)

>
> so if I get you correclty what you are proposing for your handler to
share
> it cleanly with Linux is:
>
> if(its_the_RT-cards_IRQ){
> 	do_what_is_needed()
> 	rtl_hard_enable_irq(IRQ);
> } else {
> 	rtl_global_pend_irq(IRQ);
> }
>

Yes, exactly!

If I do the above.. the system never hangs.

If I don't do the above, and unconditionally call rtl_hard_enable_irq() 
(something that lots of drivers and examples out there like to do) then 
the system will hang within a few milliseconds of loading the driver!!

(Although if I unload the linux-side drivers first there is no hang, of 
course).


>>
>> That seems to be the cause of my system lockups when sharing hard IRQs
>> with non-realtime linux drivers.  At least that's my theory.
>
> the thing that supprises me is that the timer irq is shared and there are
> no hangups - see scheduler/arch/rtl_time.c _8254_checklinuxirq, note that in
> _8254_irq rtl_hard_enable_irq(0) is called unconditionally before then
> conditionally pending it. So its not clear to me where the hangs are
> comming from.

Hmm that is surprising.  I am having a hard time coming up with an 
explanation for why this works... but perhaps the timer doesn't fire so 
quickly as to overflow the stack.

I am not really sure.. but conceptually even the situation with the timer 
is different.  In this situation one IRQ is shared for one physical 
device.  That's not the same as sharing one IRQ for many physical 
devices... I am not sure how the two situations differ, but I could 
imagine scenarios of maybe boards repeatedly 'retrying' IRQs if they don't 
get serviced in time?  (Anyone know how IRQs work in PCI and if this 
happens?  Or if some boards out there like to do this say if their buffers 
are filling up, etc..?)

I suspect when you have multiple devices.. some of them go haywire and may 
fire IRQs too quickly thus in effect never giving Linux a chance to run 
and actually service the requests.....?

>
>>
>> Is the above assessment of the situation correct or is my theory all wrong
>> and are my system lockups probably caused by other causes?
>>
> is the driver some generic hardware that I could lock up my systems with it
> here too ;) if so I'll give it a shot.
>

Hahaha, well.. my driver is actually for a very specific board, the Lynx 
22 high-performance sound board..  and the manufacturer of the board made 
me sign an NDA on everything including the driver sources I write.. so 
even if I sent you the board we would have to go through stupid legal 
hurdles for you to be able to get a copy of my driver... :/

HOWEVER.. you can try this without the board!  What you can do is create a 
'dummy' driver that simply registers a handler for a very well-known and 
heavily shared IRQ number.  On my system (a crappy VIA-kt266a based 
motherboard) every PCI board (and even some of the on-board devices) uses 
IRQ 11!!  Anyway.. see if your system has an IRQ that is shared in 
particular with network boards, then register a realtime handler for that.

Then.. in your handler do nothing, eg:

unsigned int my_dummy_handler(unsigned int irq, struct pt_regs *)
{
  rtl_global_pend_irq(irq); // this is optional but recommended
  rtl_hard_enable_irq(irq); // uncomment this to not get system hangs.. :)
  return 0;
}

And see what happens.  On my system the above is guaranteed to hang the 
system!

-Calin




_______________________________________________
Rtl mailing list
[email protected]
http://hq.fsmlabs.com/mailman/listinfo/rtl
http:/www.rtlinux-gpl.org/