Re: [PATCH] dma/swiotlb: make high watermark tracking boot-time opt-in

On 20.07.2026 20:30, Michael Kelley wrote:
> From: Frank Chen <[email protected]> Sent: Monday, July 13, 2026 8:25 PM
>> From: chenhuguanshen <[email protected]>
>>
>> Under heavy concurrent DMA traffic on CoCo VMs, inc_used_and_hiwater()
>> performs an atomic_long_add_return() plus a CAS loop on the global
>> used_hiwater, and dec_used() performs an atomic_long_sub() on total_used.
>> All CPUs contend on the same cacheline, causing measurable throughput
>> degradation at scale.
>>
>> Historically these counters were only compiled in under CONFIG_DEBUG_FS,
>> which means production kernels with debugfs paid the atomic overhead
>> unconditionally once the symbols were present. Make the tracking
>> boot-time opt-in instead so that it is disabled by default with near-zero
>> overhead via static_call, and can be enabled via "swiotlb=track_hiwater"
>> parameter on demand for debugging.
>>
>> Changes:
>>
>> - The previous inc_used_and_hiwater() maintained a separate atomic
>>   total_used counter for accurate hiwater updates. This is replaced with
>>   mem_used() which sums across areas. While less precise (no cross-area
>>   locking), the result is adequate for high watermark statistics and
>>   avoids the overhead of maintaining total_used on every alloc/free.
> I'm not sold on giving up an accurate hiwater calculation. If it's
> optional, let's have the option be to maintain total_used on every
> alloc/free so the hiwater calculation is accurate.
>
> Maintaining hiwater based on mem_used() is already pretty low
> overhead in that it only does a global atomic operation if the hiwater is
> raised. After a startup transient that establishes a typical steady-state
> hiwater, most ongoing swiotlb alloc/free operations won't raise
> the hiwater and so won't do any atomic operations to maintain it.
>
> To recap, there are three levels of hiwater maintenance:
>
> 1) None
> 2) Based on the less precise mem_used()
> 3) Based on an atomically update global total_used value
>
> The perf difference between #1 and #2 seems minimal outside of
> the initial ramp to a steady-state hiwater value. The perf impact
> of #3 can indeed be more significant, especially at high CPU counts.
>
> Having the kernel boot line offer all three is arguably overkill.
> I'd suggest offering #1 and #3 (with #1 as the default), or #2
> and #3 (with #2 as the default and !CONFIG_DEBUG_FS providing
> #1).
>
> There's another aspect here as well. Robin Murphy had 
> suggested a sysctl option, so that the accurate hiwater maintenance
> could be turned on/off at runtime, not just a boot time. My earlier
> comment about boot time was probably unclear. I was thinking in
> terms of a runtime option that could *also* be set at boot time, which
> sysctl does. But I'll admit that enabling accurate hiwater calculation
> at runtime is tricky because some swiotlb buffers may be allocated
> at the time of the enablement, and total_used could go negative
> when those are freed. For runtime sysctl enablement, would
> probably have to do stop_machine() to freeze everything, and set
> total_used to the value of mem_used() so that everything is
> correct. I haven't worked through all the details.
>
> All that said, I don't want to make too big of an issue out of this,
> and end up over-engineering with complexity that exceeds the
> value. I could be persuaded to have boot time enablement but
> not runtime enablement. I'm less happy with losing entirely
> the accurate calculation (my #3 above) that's in the current
> upstream code.

Indeed keeping the accurate measurements is important to track potential

spikes of hiwater level. However imho it would be enough to use boot time

enablement.

Best regards
-- 
Marek Szyprowski, PhD
Samsung R&D Institute Poland