[Progress Report] libgomp Optimizations for Scheduler Guided OpenMP Execution in Cloud VMs

Hello Everyone,

I am Roy, and this is my progress report on what I have been doing.

First, based on the remarks of my mentors, I had to add an eBPF cleanup
program to clean dead OpenMP thread TIDs from the omp_threads_map when the
corresponding threads exit.

Then, with the guest_ivshmem driver provided by my mentor, I created a
systemd service that builds and loads this kernel module in the guest on
each boot of a VM. Since this driver also exposes the
bpf_guest_ivshmem_g2h_write kfunc used to write whether a vCPU is running
an OpenMP thread or not, I adjusted the omp_thread_reg BPF program to
utilize it as such. I also modified the host_ivshmem driver provided by the
mentors by adding a kfunc, bpf_host_ivshmem_g2h_read, to read the
information provided by the guest eBPF programs via the IVSHMEM
guest-to-host page.

Given now that the guest is providing this information (i.e., whether a
vCPU is running an OpenMP thread or not), I now utilize it on the host in
the phantom_switch_handler BPF program to mark a vCPU as phantom only if it
is running an OpenMP thread and the vCPU is not running on the host.

I then worked on making the phantom average computed on the host available
to the guest userspace OpenMP programs. That involved adding a character
device for the guest_ivshmem driver that exposes the phantom average on the
host-to-guest page to userspace programs on the guest via file operations
such as read on the char device's device node.

Modifying the GCC code base by adding a new environment variable
GOMP_WAIT_POLICY that retains the behaviour of OMP_WAIT_POLICY but
introduces a new value pvsched for paravirtualised barrier synchronisation.
I added the parsing logic for this environment variable and modified the
do_spin function to read this phantom average while an OpenMP thread is
spinning at a barrier and, when phantoms are detected, immediately block an
OpenMP thread.

I added tests to verify the interaction between the modified libgomp and
Phantom Tracker, which involved the use of several eBPF programs:

   1.

   A uprobe on the libgomp waiting path to identify when an OpenMP thread
   enters the waiting path.
   2.

   A kprobe/kretprobe on the read operation of the guest_ivshmem character
   device to observe phantom-average reads.
   3.

   A sys_enter_futex tracepoint to observe futex syscalls made by OpenMP
   threads.
   4.

   A sched_switch tracepoint to observe when OpenMP threads are scheduled
   out.


Following the latest discussions with my mentors, I am currently working on:

   -

   Replacing the read operation of the guest_ivshmem character device with
   an mmap operation so that the IVSHMEM pages can be directly mapped into
   guest userspace. This should avoid performing a read/pread operation every
   time the phantom average is accessed.
   -

   Adapting the libgomp implementation and test to the new mmap-based
   interface.
   -

   Exposing the relevant libgomp symbols in a way that provides a reliable
   interface for attaching uprobe eBPF programs.
   -

   Adding GCC tests for the new GOMP_WAIT_POLICY functionality.

Best Regards
Nchang Roy