Re: stress-ng process termination issue

I think that I have an understanding of how this problem arises. 
Apologies for the long explanation especially if much of this is already 
apparent to you.

Thread $task61.1 receives the message to handle the SIGALRM signal for 
$task61. Signal handling requires the main thread to be aborted by 
calling thread_abort($task61.0). Thread $task61.1 then enters 
thread_sleep() awaiting thread $task61.0 to become suspended. Thread 
$task61.1 has a state of TH_WAIT|TH_RUN at this time.

The stressor program ($task60) reaches its "limit of patience" and sends 
SIGKILL to the worker ($task61). SIGKILL is handled specially and so 
$task60 calls task_terminate($task61) directly. An early phase of this 
calls task_hold_locked() which calls thread_hold() on each thread in 
$task61. This is when $task61.1 (whilst sleeping) changes to 
TH_WAIT|TH_SUSP before it emerges from thread_sleep().

When $task61.1 is woken up by thread_wakeup_prim() it matches the case 
where it is "Either already running, or suspended" so the code only 
removes the TH_WAIT state and the thread is not set running. It remains 
in this state as there are no further triggers to set it running again 
to tidy its state before termination. The most relevant state that 
cannot be tidied is the remaining reference to thread $task61.0.

Task $task60 is now spinning away with continuous attempts to terminate 
the first thread in the list ($task61.0) but that thread never 
terminates because of the reference held by $task61.1. Both $task60 and 
$task61 are now permanently stuck in their current states.

My suggestion is to alter the code within task_terminate() to call 
thread_force_terminate() for each thread in the list rather than just 
the head of the list. Then repeat the iteration through the remaining 
threads in the list until there are no threads remaining. This is 
compared to the existing code which relies on the threads being able to 
be terminated in sequence which might not always be possible (if my 
analysis above is correct).

I've attached my proposal for review. The solution appears clumsy but I 
could not see a more trivial method of traversing the list safely with 
the locks necessarily released for the call to thread_force_terminate(). 
I wouldn't be at all surprised if what I have proposed is not safe but 
I'd welcome any feedback to improve what I have or to find error in my 
analysis. I've included the original code '#if 0' for more simple 
comparison rather than a diff at this stage. In any case, even if my 
suggestion finds favour, I'd imagine this alteration would want much 
scrutiny before being released into what is a critical part of gnumach.

I have tested this patch with my stress-ng test case and it has not 
failed for more than 90 minutes now which is some kind of record. It 
usually fails within 20 minutes and often less than that.

Regards,

Mike.

On 23/07/2025 20:13, Michael Kelly wrote:

> Some additional context for consideration. The thread 0xf60f9170 has a 
> reference count of 1 so presumably other than during repetitions of 
> the while loop in $task60.0 the only reference is held by $task61.1. 
> That thread is sleeping waiting for TH_EV_WAKE_ACTIVE on thread 
> 0xf60f9170. That wakeup event presumably never arrives. Is that down 
> to the task it is associated with being terminated?
>
> On 22/07/2025 20:14, Michael Kelly wrote:
>> Hi All,
>>
>> I've been experimenting with stress-ng for some time to stress test 
>> my hurd virtual machine. This has already exposed a few problems but 
>> here is another. Sorry, for the long explanation, but it might be 
>> necessary to make sense of the problem. The scenario under test goes 
>> something like:
>>
>> 1) Top level supervisory process 'stress-ng' begins execution
>>
>> 2) It forks N times, one per stressor under test (in my case 64 
>> times). Call these processes 'stressor'.
>>
>> 3) The particular tests I am running are stress-vm and stress-mmap. 
>> In these tests each of the stressor processes forks again so that it 
>> can be supervised and restart the test should it run out of 
>> resources. Call these processes 'worker'.
>>
>> 4) Each stressor sets a timeout using alarm() and then waits for the 
>> worker to terminate by calling waitpid().
>>
>> 5) The stressor SIGALRM handler sets a variable tested occasionally 
>> within the worker. If the worker tests that variable quickly then it 
>> exits normally. If it does not, then the stressor sends a series of 
>> signals SIGALRM (4 times), SIGTERM then finally SIGKILL with a short 
>> time gap between them.
>>
>> The test scenario I set up uses all the vm's real memory and a 
>> certain portion of swap. Consequently when the timeout expires, many 
>> of the processes are paged out and they do not respond quickly which 
>> means that many workers receive all 6 signals. Occasionally, one of 
>> the stressor processes gets stuck within this while loop within 
>> task_terminate ($task60.0):
>>
>> while (!queue_empty(list)) {
>>     thread = (thread_t) queue_first(list); /* thread is 0xf60f9170 
>> and is within the worker process */
>>     ......
>>     thread_force_terminate(thread);
>>     ......
>> }
>>
>> thread_force_terminate(thread) calls thread_halt(thread, TRUE) and in 
>> this instance does very little as the the thread is already halted 
>> and it simply increases the thread suspend_count (currently standing 
>> at 0x64c0fc8e !). The thread is not removed from the list and it is 
>> repeatedly processed in the loop.
>>
>> The thread 0xf60f9170 is in $task61 (the worker) and is the main 
>> thread which does all the stress testing. Examining its state 
>> suggests it is already halted with a state of 0x112 
>> (TH_SUSP|TH_HALTED|TH_SWAPPED).
>>
>> All stack traces are attached and are annotated with extra context.
>>
>> I'm trying to make sense of the thread code but as it's rather 
>> complex I thought it might save time by asking if anyone had any 
>> input to make. In particular what do I need to look at or consider to 
>> determine why the state has ended this way? Better yet someone might 
>> immediately see the cause of the problem. I have a virtual machine 
>> snapshot of this moment saved so I can easily relay any additional 
>> information required.
>>
>> There is a 2nd thread ($task60.1) in the stressor process which is 
>> also looping but I think that is just stuck waiting for the 
>> task_terminate() to complete. (This 2nd thread is processing a 
>> secondary timeout setup by the stressor using alarm(1) but I don't 
>> think that is necessarily relevant).
>>
>> None of the threads in $task61 appear to be active based on their 
>> 'last updated' time reported by the kernel debugger.
>>
>> Any ideas?
>>
>>
>

#if 0
        while (!queue_empty(list)) {
                thread = (thread_t) queue_first(list);
                thread_reference(thread);
                task_unlock(task);
                thread_force_terminate(thread);
                thread_deallocate(thread);
                thread_block(thread_no_continuation);
                task_lock(task);
        }
#else
        while (!queue_empty(list)) {

          thread = (thread_t) queue_first(list);
          thread_reference(thread);

          do
            {
              thread_t next = (thread_t) queue_next(&thread->thread_list);

              if (!queue_end(list, (queue_entry_t) next))
                  thread_reference(next);

              task_unlock(task);
              thread_force_terminate(thread);
              thread_deallocate(thread);
              thread_block(thread_no_continuation);
              thread = next;
              task_lock(task);
            }
          while (!queue_end(list, (queue_entry_t) thread));
        }
#endif