Re: EVL: Latency issues probably connected to amdgpu driver

Hannes Diethelm <[email protected]> writes:

> Am 27.07.26 um 14:54 schrieb Philippe Gerum:
>> Hannes Diethelm <[email protected]> writes:
>>=20
>>> Hello
>>>
>>> This is not a new issue and also probably not an evl issue but I try it=
 anyway
>>> here.
>>>
>>> I have latency spikes as soon the GPU has something to do. The best way=
 to reproduce
>>> it so far is watching YouTube in Firefox where it happens withing 5-10 =
min.
>>>
>>> It is the same with Xenomai4 or PREEMT_RT while with Xenomai4, the spik=
es are
>>> somewhat smaller (~100 us vs ~150 us).
>>>
>>> If the GPU stays idle, the latency is ~1us idle / ~2us with stress-ng, =
quite good.
>>> Moving windows around / open close increases it to ~3-4us.
>>>
>>> My first theory was, that probably the copy CPU <> GPU is creating the =
issue with
>>> keeping memory or cache busy. So i cripple the PCI to Gen1x1 with:
>>>
>>> amdgpu.pcie_gen_cap=3D0x00010000 amdgpu.pcie_lane_cap=3D0x00010000
>>>
>>> Interestingly, the latency is way worse, up to 500us just moving some w=
indows around,
>>> so it might be something locking up the EVL core while copying. However=
, as much as I
>>> understand how dovetail works, this should be impossible?
>>>
>>> Do you have any hints how to debug / fix this issue?
>> Looking at the ftrace logs (thanks for this btw, that's exactly what
>> we
>> need to chase such kind of issues), I see some unexplained ~140 =C2=B5s
>> latency bump right after programming the LAPIC deadline timer.
>> test2_all:
>>            <idle>-0       [003] *.~1.   834.920082: lapic_next_deadline =
<-evl_program_proxy_tick
>>            <idle>-0       [003] *.~1.   834.920083: do_trace_write_msr <=
-lapic_next_deadline
>>            <idle>-0       [003] *.~1.   834.920083: write_msr: 6e0, valu=
e 278dc7d4cab
>>            <idle>-0       [003] *.~1.   834.920222: evl_timer_shot: latm=
us_pulse_handler at 834.920317 (delay: 94 us, 37780 cycles)
>> Since there is not reason for such MSR update to cause such latency
>> peak
>> and no smoking gun is visible from any CPU around this spot either, I'd
>> suspect an issue with some kind of power management activity. AMD GPUs
>> have pstates, any luck with any tweak mentioned by [1]?
>> [1] https://docs.kernel.org/admin-guide/pm/amd-pstate.html
>>=20
>
> [1] is for amd cpu's. [2] would be the GPU
>

Yep, confusing typo. The idea was that we might have some pstate
shenanigan occurring as the CPU stalls, waiting for the GPU to complete
some batch.

> By using intel_idle.max_cstate=3D1 cpufreq.off=3D1, most CPU power saving=
 should be disabled. But
> I might have missed a flag. powertop shows constant frequency and only C0=
/C1 activity.
>

Ok, so the pstate assumption does not hold.

> However, I would expect that CPU power saving generates spikes
> depending on CPU activity. But I have spikes depending on the GPU activit=
y.
>

IMHO, the way GPUs and CPUs may interact via the interconnect makes it
difficult to precisely assess which is to be charged for the latency.

> Even moving the mouse around on the desktop is immediately visible in the=
 latency. Just only
> increasing it by ~1-2us while stress-ng stressing the CPU has less influe=
nce.
>
> I did some more tests with
> amdgpu.pcie_gen_cap=3D0x00010000 amdgpu.pcie_lane_cap=3D0x00010000
> This cap's the PCIE speed for the GPU. Interestingly, if I add these flag=
s,
> latency is way worse. Moving the mouse can generate ~200us delays. Watchi=
ng YouTube,
> its up to 500us.
>
> Without these options:
> LnkSta:	Speed 8GT/s, Width x8
>
> With them:
> LnkSta:	Speed 2.5GT/s (downgraded), Width x1 (downgraded)
>
> It feels like: GPU needs to copy some data -> freeze whole CPU and copy -=
> done
>

This seems to point to a CPU stall, waiting for the GPU to complete some
task.

> Attached the results, this time with 1ms period so I don't get overruns.
>
> test10 with evl trace -eirq
> test11 with evl trace -e
>
> [2] https://www.kernel.org/doc/html/v4.20/gpu/amdgpu.html
>
>
> [2. application/x-xz; testresults2.tar.xz]...

          <idle>-0       [003] *.~1.   513.339298: evl_timer_shot: latmus_p=
ulse_handler at 513.340060 (delay: 763 us, 304611 cycles)
          ...
          /* CPU3 enters idle state, so hard irqs are on */
          <idle>-0       [003] *..1.   513.338085: cpu_idle: state=3D1 cpu_=
id=3D3
          ...
          <idle>-0       [003] *..1.   513.339197: handle_irq_pipelined_pre=
pare <-arch_pipeline_entry
          <idle>-0       [003] *.~1.   513.339239: arch_handle_irq <-arch_p=
ipeline_entry
          <idle>-0       [003] *.~1.   513.339269: irq_to_desc <-arch_handl=
e_irq
=20=20=20=20=20=20=20=20=20=20
Yep, at any rate, the traces clearly point at an hardware related stall.

--=20
Philippe.