[PATCH 0/4] unmap_page_range optimisation

Previous discussion:
RFC: unmap_page_range optimisation (avoiding emulation faults during VM migration)
https://lore.kernel.org/xen-devel/[email protected]/

This series adds a new pte_get_and_clear hypercall which Linux can use
to significantly improve the performance of unmap_page_range.

The new hypercall reuses existing code from do_mmu_update, mod_l1_entry
and update_intpte but is careful not to impact performance of the
non-pte-get-and-clear code paths.

A microbenchmark[1] which allocates a large number of pages and then clears
them shows a performance increase from 1100ms to 700ms using the
new pte_get_and_clear hypercall.

Further profiling the microbenchmark with bpftrace[2] shows that Linux calls
unmap_page_range() 23 times with an average execution time of 50ms
dropping to 30ms when using the new hypercall.

[1] microbenchmark
#include <err.h>
#include <sys/mman.h>
#include <time.h>
#include <stdint.h>
#include <stdio.h>
#include <unistd.h>

static uint64_t nsec(void)
{
    struct timespec ts;
    clock_gettime(CLOCK_MONOTONIC, &ts);
    return (uint64_t)ts.tv_sec * 1e9 + ts.tv_nsec;
}

int main(int argc, char **argv)
{
    const size_t len = 1024UL * 1024 * 1024 * 4;
    const long pagesz = sysconf(_SC_PAGESIZE);

    char *p = mmap(NULL, len,
                   PROT_READ | PROT_WRITE,
                   MAP_PRIVATE | MAP_ANONYMOUS,
                   -1, 0);
    if ( p == MAP_FAILED )
        err(1, "mmap");

    /* Fault every page in */
    for (size_t i = 0; i < len; i += pagesz)
        p[i] = 1;

    uint64_t start = nsec();

    if ( madvise(p, len, MADV_DONTNEED) )
        err(1, "madvise");

    uint64_t end = nsec();

    printf("MADV_DONTNEED on %zu MB took %.3f ms\n",
           len / 1024 / 1024,
           (end - start) / 1e6);

    munmap(p, len);
    return 0;
}

[2] bpftrace
bpftrace -e '
kprobe:unmap_page_range
/comm == "a.out"/
{
    @start[tid] = nsecs;
}

kretprobe:unmap_page_range
/@start[tid] && comm == "a.out"/
{
    $delta = nsecs - @start[tid];
    @count = count();
    @total = sum($delta);
    delete(@start[tid]);
}

interval:s:10
{
    printf("avg call time = %d ns (%d calls)\n",
           (@total / @count), (uint64)@count);
    exit();
}'

Kevin Lampis (4):
  x86: extend update_intpte() to support atomic get-and-update
  x86: extend mod_l1_entry() to optionally return the old PTE value
  x86: extend do_mmu_update() to support returning the old PTE value
  x86: add new pte_get_and_clear hypercall

 xen/arch/x86/mm.c            | 106 +++++++++++++++++++++++++++++------
 xen/arch/x86/pv/mm.h         |  29 ++++++----
 xen/include/hypercall-defs.c |   2 +
 xen/include/public/xen.h     |   1 +
 4 files changed, 109 insertions(+), 29 deletions(-)

-- 
2.52.0