cannot run trace-cmd split in parallel

Hi!

I've been having this very strange issue for as long as I've been using "trace-cmd split". Actually I didn't write about it until now because it's so strange I was sure the blame is on the user 😊

When I use "trace-cmd split" in parallel, I get randomly invalid output. This happens specifically when I use the start/end arguments.

To reproduce, take any trace.dat (as far as I can tell), and run the following command, which is 5 nearly identical command lines run in parallel. The only difference is in the start/end arguments. Without this difference, the issue does not reproduce.

trace-cmd split -i trace.dat -o /tmp/out1 <start> <end> & trace-cmd split -i trace.dat -o /tmp/out2 <start+1> <end+1> & trace-cmd split -i trace.dat -o /tmp/out3 <start+2> <end+2> & trace-cmd split -i trace.dat -o /tmp/out4 <start+3> <end+3> & trace-cmd split -i trace.dat -o /tmp/out5 <start> <end>

If you compare the output of the first and last command, which are in bold and you can see they are the exact same, the output is different.
diff /tmp/out1.1 /tmp/out5.1

But it's not consistent, every run will behave differently, so you might need few runs to get this. It might also not happen at all, I guess. Statistics.

You can expect the diff to be different if you see this in the stdout/stderr:
…
libtracecmd: No such file or directory
  can not stat '/tmp/.tmp.tmp.0'
trace-cmd: No such file or directory
  Failed to append tracing data

libtracecmd: No such file or directoryש
  can not stat '/tmp/.tmp.tmp.0'
trace-cmd: No such file or directory
  Failed to append tracing data

It looks like as if trace-cmd is using some file to store temporary data, and the same filename is used by all processes.

Can anyone help me understand this weird behavior?

Thanks!
Sharon