RE: EtherCAT on PREEMPT_RT: send-recv tail ~5 0-80μs, looking for guidance

 
 hi,

By the past, on some rtl8139 chips,I had problems where  there was a register to program irq delays .
I mean the irq signal was triggered after this delay by the chip.
Check if there is similar register on this chip.

Regards
Steph


________________________________________
De: Alexander Dahl <[email protected]>
Envoyé: Vendredi 13 mars 2026 11:15
À: 김민수 <[email protected]>
Cc: [email protected] <[email protected]>
Objet: Re: EtherCAT on PREEMPT_RT: send-recv tail ~50-80μs, looking for guidance 


        External Sender: Use caution with links/attachments.

Expediteur externe : Soyez prudent avec les liens/pieces jointes.



Hei hei,



Am Fri, Jan 30, 2026 at 02:33:58PM +0900 schrieb 김민수:

> To: [email protected]

> Subject: EtherCAT on PREEMPT_RT: send-recv tail ~50-80μs, looking for guidance

>

> --

> Hi all,

>

> I'm working on reducing worst-case EtherCAT send-recv round-trip

> latency on PREEMPT_RT 6.8-rt8, targeting under 50μs. I've done

> ftrace analysis over ~50k cycles and tried two kernel-side fixes,

> but the measured tail (50-80μs) exceeds what my trace can account

> for (~31μs). I'd appreciate help identifying what I'm missing.

>

> == Environment ==

>

>   - Kernel: 6.8.0-rt8 (PREEMPT_RT)

>   - CPU: Intel, core 3 isolated

>     - isolcpus=domain,managed,3 nohz_full=3 rcu_nocbs=3

>     - intel_pstate=disable idle=poll intel_idle.max_cstate=0

>   - NIC: RTL8168H (r8169 driver), IRQ pinned to CPU3

>     - Coalescing off: rx-usecs=0, rx-frames=1

>   - EtherCAT master: SOEM (Open EtherCATsociety)

>     - SCHED_FIFO 99, pinned to CPU3, 1ms cycle

>     - AF_PACKET raw socket (ETH_P_ECAT)

>     - Measures send-recv with clock_gettime(CLOCK_MONOTONIC)

>

> == What I've tried ==

>

> Using ftrace (function_graph + sched_switch) on CPU3:

>

> 1) ksoftirqd preemption during NAPI poll

>

>    During rtl8169_poll() → napi_gro_receive() → sock_def_readable()

>    → wake_up(), SOEM (FIFO 99) preempts ksoftirqd/3 (CFS) mid-poll.

>    SOEM then blocks on rt_spin_lock (socket wq_head->lock, held by

>    wake_up) → D state ~2μs → returns to ksoftirqd → preempts again

>    after recv() completes. Measured overhead: 5.5μs mean, up to 25μs.

>

>    Fix: chrt -f -p 99 ksoftirqd/3

>

>    Result: D state eliminated, recv() peak improved ~3μs, but

>    max got worse (77μs → 88μs).

>

> 2) TX completion IRQ elimination

>

>    Each cycle generates two polls: TX IRQ triggers poll#1 (rtl_tx

>    cleanup only, ~1.3μs), then RX IRQ triggers poll#2. Gap: 3-13μs.

>

>    Fix: Masked TxOK from interrupt enable register.

>

>    Result: recv() distribution narrowed, but send-recv unchanged

>    (TX poll + gap overlaps with wire delay).

>

>    Combined: No improvement beyond H1 alone.



If you see latencies due to ksoftirqd, I assume you don't use threaded

NAPI?  Did you try?



Quoting from my own notes here:



    NAPI processing mostly happens in eth softirq threads, but not always,

    especially with higher system load it might end up in ksoftirqd, which

    runs with low priority, and which process priority is not recommended to

    be changed (with RT).



    Link: https://lwn.net/Articles/833840/

    Link: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=5fdd2f0e5c64

    Link: https://lwn.net/Articles/853289/



What we did:



+           # enable napi threads and set realtime priority

+           echo 1 > /sys/class/net/eth0/threaded

+           sleep 1

+           pgrep napi/eth0 | xargs -n 1 chrt --pid 49



That eliminated latency spikes for us, on completely different

hardware though. ^^



Greets

Alex