[Bug 280037] KTLS with Intel QAT may trigger kernel panics

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D280037

--- Comment #22 from ss3bsd <[email protected]> ---
Created attachment 272912
  --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=3D272912&action=
=3Dedit
draft patch [B]

For reference, I have been using QAT stably since then under the following
conditions:
  - with a slightly refined patch [A]
  - kern.ipc.tls.cbc_enable: 0
  - do not disable or unload qat after its activation
  - used for KTLS, OpenSSL qatprovider (TLS handshake), IPsec, and
(infrequently) GELI
  - upgraded to 15.1 from 14.x when 15.1 was released

Note:
  - Suspending the system is outside the scope of my use case.
  - The patch uploaded in Comment 17 is not required for the stability if p=
atch
[A] is applied. Instead, for performance-wise (off-topic), I am using anoth=
er
patch [B] that launches a higher number of ktls worker threads than the num=
ber
of cpu cores.


Background (my observation) of the patches:
  The KTLS implementation (FreeBSD 13.x) used to use the QAT device
synchronously both for receive (RX) and send (TX) sides. It now (14.0 and
later) uses asynchronous dispatch for TX (RX is kept synchronous).=20

  [A] The asynchronous dispatch improved performance but seems to have
introduced some instability such that a ktls session hangs or memory leaks =
if
the queue for asynchronous dispatch is full or (IIRC) a kernel panic occurs=
 if
a ktls session is terminated (freed) while a command remains in the queue. =
This
patch is intended to prevent those issues (and introduce some sysctl parame=
ters
that I used to switch software/hardware engine or synchronous/asynchronous
dispatch for analysis).

  [B] While the ktls implementation synchronously works with the engine for=
 RX,
the number of ktls worker thread is limited to the number of cpu cores. This
may not be ideal if a hardware offload engine (e.g. qat) is used. If the nu=
mber
of cpu cores is fewer, not enough number of commands are parallelly submitt=
ed,
then the engine is not fully utilized. This patch introduces a sysctl param=
eter
kern.ipc.tls.thread_overcommit by which 4 times as many ktls worker threads=
 can
be created. (In my environment, the patch improved the benchmark (parallel =
ktls
wget) result of receive throughput from 5-6 Gbps to almost 10 Gbps for 10Gb=
E.)

--=20
You are receiving this mail because:
You are the assignee for the bug.=