Re: EVL: Latency issues probably connected to amdgpu driver

Hannes Diethelm <[email protected]> writes:

> Hello
>
> This is not a new issue and also probably not an evl issue but I try it a=
nyway
> here.
>
> I have latency spikes as soon the GPU has something to do. The best way t=
o reproduce
> it so far is watching YouTube in Firefox where it happens withing 5-10 mi=
n.
>
> It is the same with Xenomai4 or PREEMT_RT while with Xenomai4, the spikes=
 are
> somewhat smaller (~100 us vs ~150 us).
>
> If the GPU stays idle, the latency is ~1us idle / ~2us with stress-ng, qu=
ite good.
> Moving windows around / open close increases it to ~3-4us.
>
> My first theory was, that probably the copy CPU <> GPU is creating the is=
sue with
> keeping memory or cache busy. So i cripple the PCI to Gen1x1 with:
>
> amdgpu.pcie_gen_cap=3D0x00010000 amdgpu.pcie_lane_cap=3D0x00010000
>
> Interestingly, the latency is way worse, up to 500us just moving some win=
dows around,
> so it might be something locking up the EVL core while copying. However, =
as much as I
> understand how dovetail works, this should be impossible?
>
> Do you have any hints how to debug / fix this issue?

Looking at the ftrace logs (thanks for this btw, that's exactly what we
need to chase such kind of issues), I see some unexplained ~140 =C2=B5s
latency bump right after programming the LAPIC deadline timer.

test2_all:
          <idle>-0       [003] *.~1.   834.920082: lapic_next_deadline <-ev=
l_program_proxy_tick
          <idle>-0       [003] *.~1.   834.920083: do_trace_write_msr <-lap=
ic_next_deadline
          <idle>-0       [003] *.~1.   834.920083: write_msr: 6e0, value 27=
8dc7d4cab
          <idle>-0       [003] *.~1.   834.920222: evl_timer_shot: latmus_p=
ulse_handler at 834.920317 (delay: 94 us, 37780 cycles)

Since there is not reason for such MSR update to cause such latency peak
and no smoking gun is visible from any CPU around this spot either, I'd
suspect an issue with some kind of power management activity. AMD GPUs
have pstates, any luck with any tweak mentioned by [1]?

[1] https://docs.kernel.org/admin-guide/pm/amd-pstate.html

--=20
Philippe.