Debian-13 bad performance on IPC

Sorry for the lengthy message, but I don't know how to better compress
the facts...

- I have a data aquisition device which delivers data with ~2.8MB/s to
  an IPC (fanless Industrial PC, Intel core i7-1185G7E).

- When receiving a START signal, data are stored to SSD, and an analysis
  program is started.  The analysis program in turn starts several
  sub-processes, which in turn start several sub-sub-processes.
  All sub-sub-processes read the recorded data files in a "tail -f" fashion.

- When receiving the STOP signal after 5secs, recording is stopped and
  the analysis result is collected from the sub-sub-processes.


* Crucial point is the time it takes from STOP to RESULT-RECEIVED:


We started on Windows-11:
- IPC, Windows-11 with all updates
  (11th Gen Intel(R) Core(TM) i7-1185G7E @ 2.80GHz, 8 logical processors)
  
 Result-recv[s]  Count
 ----------------------
	2.20     2
	2.25     82
	2.30     328
	2.35     767
	2.40     857
	2.45     632
	2.50     372
	2.55     136
	2.60     40
	2.65     16
	2.70     1
	2.80     1

 =>  Approx. 2.4s to "Result-received", with some variation,
 but fairly constant.  So far so good.

The idea now was that Linux/Debian-13 should perform better on
identical hardware.


- the very same IPC, Debian-13, with all patches
  (11th Gen Intel(R) Core(TM) i7-1185G7E @ 2.80GHz, 8 logical processors)
 Result-recv[s]  Count
 ---------------------- 
	0.60     1
	0.65     75
	0.70     137
	0.75     109
	0.80     21
	0.85     8
	0.90     29
	0.95     18
	1.00     5
	1.45     1
	1.50     1
	1.55     2
	1.60     10
	1.65     17
	1.70     27
	1.75     177
	1.80     532
	1.85     384
	1.90     92
	1.95     39
	2.00     15
	2.45     1
	2.50     1
	2.55     4
	2.60     6
	2.65     102
	2.70     365
	2.75     395
	2.80     604
	2.85     562
	2.90     115
	2.95     19
	3.00     7
	3.45     3
	3.50     9
	3.55     12
	3.60     17
	3.65     15
	3.70     21
	3.75     31
	3.80     39
	3.85     21
	3.90     7
	3.95     1

 => There are several clusters of answer times, all ~1s apart.
 => the majority of answers are *later* than on Windows

I don't understand
- why the performance varies that much,
- why the performance is worse than on Windows on the very same PC hardware.

I have tried:
- echo performance | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
  - no difference
- sudo powerprofilesctl set performance
  - no difference
- cpupower-gui: set performance mode, limit min-CPU-Freq to 2500
  - no difference
- intel_pstate=disabled at boot
  - performance even worse
- intel_pstate=passive at boot, cpu-governor schedutil
  - slightly worse performance
- Debian-12, same software
  - no difference



For comparison, the very same program on my Desktop PC (with
cpu-governor 'schedutil', since there is a different kernel module at work)
- Desktop PC, Debian-13 with all patches
  (Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz, 8 logical processors)
 Result-recv[s]  Count
	0.40     2
	0.45     1
	0.50     1
	0.55     4
	0.60     14
	0.65     21
	0.70     46
	0.75     68
	0.80     82
	0.85     55
	0.90     16
	0.95     8
	1.00     3
	1.05     1

This is what I had expected on the IPC as well.


The problem seems to be that some of the sub-sub-processes are started
very late, not until STOP is received:

	local_time[ms]  what
	----------------------
	17817764 68904   MAIN "START" sub-1 started
		 
	17817764 69213   sub-1: OPENED sub-1-1
	17817764 69215   sub-1: OPENED sub-1-2
	17817764 69216   sub-1: OPENED SUB-1-3
		 
	17817764 71455   sub-1-2: opened sub-1-2-1
	17817764 71455   sub-1-2: opened sub-1-2-2
		 
	17817764 73451   sub-1-3: opened sub-1-3-1
	17817764 73452   sub-1-3: opened sub-1-3-2

	17817764 73905   MAIN "STOP" sub-1 close, wait for result

    Note: NO "sub-1-1 OPENED" yet!

	17817764 74068   sub-1-3: finished sub-1-3-1
	17817764 74072   sub-1-2: finished sub-1-2-1
	17817764 74076   sub-1-3: finished sub-1-3-2
	17817764 74081   sub-1-2: finished sub-1-3-2
		 
	17817764 74165   sub-1: CLOSED sub-1-2
	17817764 74166   sub-1: CLOSED sub-1-3

    Only now arrive the  "sub-1-1: opened..." !  It looks like the
    scheduler postponed running sub-1-1 until now...

	17817764 74457   sub-1-1: opened sub1-1-1
	17817764 74457   sub-1-1: opened ...
	17817764 74457   sub-1-1: opened sub1-1-9
		 
	17817764 74484   sub-1-1: closed sub1-1-1
	17817764 75737   sub-1-1: closed sub1-1-2
	...	 
	17817764 75739   sub-1-1: closed sub1-1-9
	17817764 76657   sub-1: CLOSED sub-1-1

Which program starts 'late' varies, sometimes it is sub-1-1, sometimes
sub-1-2, sometimes sub-1-3.  Sometimes all three start early, this is
when the low response times are achieved.

On my Desktop-PC and on Windows on the same hardware:
- all sub-programs (sub and sub-sub) start immediately after START
- all sub-programs are done quickly after STOP (since processing time
  is less than recording time)


So I would be grateful for any hint on how to make the IPC under
Debian-13 as performant (or better) as under Windows, at best as
perfomant as my ancient desktop PC.

TNX
R'