Re: [Question] Possible tlb_sync hang when remote SFENCE.VMA races with HSM hart_stop

On Fri, Jul 3, 2026 at 1:52 PM <[email protected]> wrote:
>
> Hello OpenSBI maintainers,
>
> We are seeing a hang during aging/stress testing and would like to check whether our analysis is correct.
>
> Environment:
> - OpenSBI base version: v1.8.1
> - Base commit in our tree: 74434f255873d74e56cc50aa762d1caf24c099f8
> - Downstream tree with platform-specific HSM hart_stop support
> - Platform: downstream Ingenic platform
>
> Observed symptom:
> When one hart executes remote SFENCE.VMA while another hart is stopping, the sender can spin forever in tlb_sync() waiting for a completion that never arrives.
>
> Our current understanding of the race is:
>
> 1. The source hart calls sbi_tlb_request() and then sbi_ipi_send_many().
> 2. sbi_ipi_send_many() builds the target mask using sbi_hsm_hart_interruptible_mask(), so a target hart can still be selected while it is in SBI_HSM_STATE_STARTED.
> 3. Before the source completes the send path, the target hart can execute sbi_hsm_hart_stop() and transition from STARTED to STOP_PENDING.
> 4. The source still queues the TLB request in tlb_update() and increments its local tlb_sync counter.
> 5. That counter is decremented only when the target hart later runs tlb_entry_process().
> 6. If the target hart has already passed the point in the stop path where pending IPIs are drained, the queued TLB request is never processed, so the source remains in tlb_sync() forever.
>
> So this looks like a race between:
> - the interruptible-mask snapshot in sbi_ipi_send_many(), and
> - the STARTED -> STOP_PENDING transition in sbi_hsm_hart_stop()
>
> In other words, tlb_sync itself does not appear to be wrong; the problem seems to be that a hart can be counted as a target, but then stop before it can process the queued TLB request.
>
> To make the window easier to hit during debugging, we temporarily added an artificial delay in sbi_ipi_send_many() in our downstream tree. However, our understanding is that the delay only increases the reproduction probability, and the race exists in principle even without that modification.
>
> Questions:
> 1. Does the analysis above look correct?

Yes, this is correct.

> 2. Is this race already known?

No, this was not know.

> 3. Is there an existing or preferred way to serialize remote rfence/IPI send against hart_stop?

My suggestion is to call sbi_ipi_process() just before
sbi_platform_early_exit() in sbi_exit(). This way the HART which is
being stopped will process the pending IPIs before going down.

> 4. Could this be related to issue #402 ("Lifelock condition under high load with many TLB shootdowns"), or is it better treated as a separate issue?

This is not related to any Livelock condition.

Regards,
Anup

-- 
opensbi mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/opensbi