libports and interrupted RPCs

I continue to use the same stress-ng test case and the only system 
'freeze' that I now see regularly (on non-rumpdisk systems) is triggered 
during processing of system call interruptions. This arises when the 
stress-ng processes near termination and receive signals. The test case 
generates a lot of interaction between stress-ng and ext2fs and many of 
the processes have an outstanding RPC at the time the signal arrives.

In such cases, the signal handling can send an 'interrupt_operation' RPC 
to the same port that the stress-ng sent its RPC request and the ext2fs 
server starts a new thread to process this additional request. ext2fs 
then calls hurd_thread_cancel() for each thread that is processing an 
RPC received on that port. But that in turn can find active RPCs sent to 
other servers (eg. storeio) which are also interrupted with an 
'interrupt_operation'.

This sequence of hurd_thread_cancel() calls all occur whilst a single 
process wide mutex is held locked (see libports:interrupt_rpcs.c). The 
same lock is also required to begin or end other RPCs on other ports and 
so they must wait until the initial interrupt_operation completes. That 
takes such a comparatively long time that sometimes so many threads 
throughout the system wait so long that a kind of system deadlock occurs 
as they are all wanting service (mostly page in from external memory 
objects) from the same task (ext2fs).

I think that the calls to hurd_thread_cancel() need to be moved outside 
of the mutex_locked regions. Doing that does potentially change the 
timing of events compared to the existing code but there can be no 
certainty about when an interrupt_operation might arrive anyway. For 
example, the RPC it should be interrupting might already have completed 
by the time the interrupt_operation RPC has been received. I don't think 
that any synchronisation is required to preserve the current processing 
order. The main changed required is to add new locking logic to safely 
coordinate access to the rpc_info structures after the locking has been 
released and reacquired.

I have a partial solution to this which I implemented hastily. I didn't 
want to spend too much time before I knew whether it was very effective 
or not. It does seem to be very effective as my test case ran without 
fault for over 20 hours before I stopped it (I don't think I've achieved 
8 hours before). Provided that nobody screams in horror at this 
approach, I'll now implement the solution properly and retest. There are 
about 5 or 6 similar cases where all RPCs on a port are cancelled within 
the global lock so they might need altering too (but they don't figure 
in this test case seemingly).