Xen Security Advisory 153 (CVE-2015-7972) - x86: populate-on-demand balloon size inaccuracy can crash guests

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

            Xen Security Advisory CVE-2015-7972 / XSA-153
                              version 3

     x86: populate-on-demand balloon size inaccuracy can crash guests

UPDATES IN VERSION 3
====================

Public release.

ISSUE DESCRIPTION
=================

The design of the memory populate-on-demand (PoD) system requires that
a guest's memory ballooning driver reach its memory reduction target.
The target is not entirely well-defined in terms of the information
visible to the appropriate parts of the system, so some unknown set of
guests (but probably most guests) will fail this criterion.

If the guest memory balloon driver does not free sufficient memory to
reach its target, the guest will proceed to run with a nonzero number
of outstanding PoD pages.  When the guest or management toolstack
touches such a page, the hypervisor would search the guest memory for
a page containing only zeroes.

If no such page is found, the guest crashes.  Prior to the patch for
XSA-150, the search might lock up the relevant physical cpu for a
while.  After the patch to XSA-150, it might crash the guest even if a
suitable zero page is available.

This means that in the current arrangements toolstack software must
apply an adjustment to a guest's PoD target as supplied to Xen.
Neither xend nor libxl do this.

IMPACT
======

Guests configured with PoD might be unstable, especially under load.

In an affected guest, an unprivileged guest user might be able to
cause a guest crash, perhaps simply by applying load so as to cause
heavy memory pressure within the guest.

This problem also allows an unprivileged guest user to exercise the
separate vulnerability described in XSA-150: an unprivileged guest
user might be able to cause a denial of service affecting the host.

VULNERABLE SYSTEMS
==================

The vulnerability is restricted to HVM guests which have been
constructed in Populate-on-Demand mode (ie, with memory < maxmem).

ARM is not vulnerable.  x86 PV VMs are not vulnerable.  x86 HVM
domains without PoD (ie started with memory==maxmem, or without
mentioning "maxmem" in the guest config file) are not vulnerable.

Systems using libxl (whether via xl, or libvirt, or another higher
layer) or xend (whether via xm, or libvirt, or another higher layer)
are vulnerable.

If the system has been stress-tested (by imposing memory load on the
guest) and found to be stable, it is less likely that the guest is
vulnerable.

Combinations of Xen, guest, guest balloon driver, and toolstack
software, which have an empirical adjustment as described in the
Description, and which have been formally stress-tested in PoD mode,
are less likely to be vulnerable.

Migration is not capable of creating a guest with outstanding PoD.  So
migrating a guest which is vulnerable might crash it.  However, if a
guest has been migrated successfully since it booted, it is no longer
vulnerable.

Xen versions back to 3.4.x are affected.

Vulnerability of a particular guest can be tested by the host
administrator using the utility `xsa153-check.c', attached to this
advisory.


MITIGATION
==========

Reducing the guest's memory target, after guest startup, can cause the
guest's ballon driver to eliminate the PoD discrepancy.  If the guest
successfully balloons down, it will no longer be vulnerable.

On systems using libxl this can be done with `xl mem-set', during or
after each guest boot:

   # ./xsa153-check `xl domid name-of-guest`
   checked domain 621 for XSA-153: VULNERABLE (1 more outstanding pages)
   try using   xl mem-set   to reduce its memory by 1 (Mby)
   or perhaps reduce /local/domain/621/memory/target by 4
   # xl list name-of-guest
   Name                  ID   Mem VCPUs      State   Time(s)
   name-of-guest        621   512     2     r-----     156.9
   # xl mem-set name-of-guest 511
   #
   [ wait for guest to give up memory ]
   # ./xsa153-check `xl domid name-of-guest`
   checked domain 621 for XSA-153: NOT vulnerable
   #

Alternatively, no matter the toolstack, it is possible for a host
administrator to bypass the toolstack code and give ballooning
instructions directly to the guest:

   [ suppose guest domid is 616, eg from xl domid name-of-guest  ]
   # ./xsa153-check 616
   checked domain 616 for XSA-153: VULNERABLE (1 more outstanding pages)
   try using   xl mem-set   to reduce its memory by 1 (Mby)
   or perhaps reduce /local/domain/616/memory/target by 4
   # xenstore-read /local/domain/616/memory/target
   520188
   # xenstore-write /local/domain/616/memory/target 520184
   #
   [ wait for guest to give up memory ]
   # ./xsa153-check `xl domid name-of-guest`
   checked domain 616 for XSA-153: NOT vulnerable
   #

The memory/target value is in decimal, and is a number of kilobytes;
it must be a multiple of 4, since a page is 4 Kb on affected systems.
The value to write should be some amount less than the value read.


It is not currently known whether use of the VM memory event
inspection facilities (in-tree, this means the xc_monitor utility)
might invalidate the workaround.


Note that guests may become unstable if given too little memory, so
large reductions of the memory target should be applied with caution,
if at all.  The expected offset related to XSA-153 is small (tens of
pages, perhaps).  If a large reduction is required, it is more likely
that either the guest is still booting up (and still working to reduce
the PoD memory), or that the guest's balloon driver is not
functioning:

   # ./xsa153-check `xl domid name-of-guest`
   checked domain 623 for XSA-153: VULNERABLE (65536 more outstanding pages)
   difference is >1Mby
   ballon driver not running or guest still booting?
   #

A guest without a working balloon driver will be unstable in PoD mode,
especially under memory pressure; this is an inherent feature of the
design of PoD.


RESOLUTION
==========

The attached patch fixes the problem for systems using libxl (via xl,
or via libvirt, or another higher layer).  At the time of writing
there is no patch for xend-based systems.

xsa153-libxl.patch            xen-unstable, Xen 4.5, Xen 4.6
xsa153-libxl.patch            Xen 4.1 to 4.4 inclusive, using libxl

(Xend was removed in Xen 4.5; so the libxl-only patch is always
sufficient for Xen 4.5 and later.)

$ sha256sum xsa153*
633df5d970af49476c2d279e604150c444834bb906f6568070f0c2e0ceaa3af4  xsa153-check.c
f5cbc98cba758e10da0a01d9379012ec56b98a85a92bfeb0c6b8132d4b91ce77  xsa153-libxl.patch
$

DEPLOYMENT DURING EMBARGO
=========================

Deployment of the patches and/or mitigations described above (or
others which are substantially similar) is permitted during the
embargo, even on public-facing systems with untrusted guest users and
administrators.

But: Distribution of updated software is prohibited (except to other
members of the predisclosure list).

Predisclosure list members who wish to deploy significantly different
patches and/or mitigations, please contact the Xen Project Security
Team.


(Note: this during-embargo deployment notice is retained in
post-embargo publicly released Xen Project advisories, even though it
is then no longer applicable.  This is to enable the community to have
oversight of the Xen Project Security Team's decisionmaking.)

For more information about permissible uses of embargoed information,
consult the Xen Project community's agreed Security Policy:
  http://www.xenproject.org/security-policy.html


NOTE REGARDING SHORT EMBARGO
============================

This issue was quickly encountered by the Security Team during our
investigations of the scope and impact of XSA-150; this issue was
originally discussed in the `Incomplete Information' section of
XSA-150 v1.  Accordingly XSA-153 is embargoed and the embargo will
end at the same time as that of XSA-150.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)

iQEcBAEBAgAGBQJWMgofAAoJEIP+FMlX6CvZaqUIAIzgbftJMwo2ywcWycAGzeDS
5iseCaCqx1OD8a00m+WvXTLX/yKKJQrgTJkDlJfgqEb4Y2NoVRUKShApSHsbFrFa
qeocl7ipBdXTYk0FZZrsBd/aCjQgL/NlYf0BCaV+tpPuehOBgJwWpIf4RltOQVkv
MxfRCGee52yUbWyFykmlKK3fxfGi4wXfMGN6zS9FXpudIBxjedRS4gyksERusXS7
hcRNEcLNzeQA+4PUmpkOzwS/NrtWiIU265kaHFsMUO8HbxcFgzFJ+15G0GK8JgY5
9XE0XWxn/B5Uc7IMiDxcFYT79C87XXvH4ctFArN9MJqss/ko0H25fz+Te8iWigc=
=vBPN
-----END PGP SIGNATURE-----

_______________________________________________
Xen-announce mailing list
[email protected]
http://lists.xen.org/xen-announce

/*
 * Program to test whether a domain is potentially vulnerable to
 * XSA-153.
 *
 * Build and run:
 *   gcc -Wall xsa153-check.c -lxenctrl
 *   ./a.out `xl domid NAME-OF-YOUR-GUEST-DOMAIN`
 *
 * For building against a built Xen source tree, rather than installed
 * headers and libraries:
 *   gcc -Wall -I ../xen.git/dist/install/usr/include/ xsa153-check.c -L ../xen.git/dist/install/usr/lib/ -lxenctrl
 *
 * Xen 4.0 and earlier lack xc_domain_get_pod_target, so this utility
 * can only be built against Xen 4.1 and later.
 *
 * IMPORTANT: Read the notes in advisory-153.txt to understand meaning
 * of the output!
 */

#include <xenctrl.h>
#include <stdlib.h>
#include <stdio.h>

#ifdef XENCTRL_HAS_XC_INTERFACE
static xc_interface *xch;
#define BAD_XCH (xch == NULL)
#define OPENXCH xc_interface_open(0,0,0)
#else
static int xch;
#define BAD_XCH (xch <= 0)
#define OPENXCH xc_interface_open()
#endif

int main(int argc, const char **argv) {
    int domid, estatus, r;
    uint64_t tot_pages, pod_cache_pages, pod_entries;

    if (argc!=2 || !(domid = atoi(argv[1]))) {
        fputs("bad usage\n",stderr);
        exit(-1);
    }

    xch = OPENXCH;
    if (BAD_XCH) {
        perror("xc_interface_open");
        exit(-1);
    }

    r = xc_domain_get_pod_target(xch, domid,
                                 &tot_pages,
                                 &pod_cache_pages,
                                 &pod_entries);
    if (r) {
        perror("xc_domain_get_pod_target");
        exit(-1);
    }

    printf("checked domain %d for XSA-153: ", domid);
    if (pod_cache_pages < pod_entries) {
        uint64_t difference = pod_entries - pod_cache_pages;
        estatus = 1;
        printf("VULNERABLE (%lu more outstanding pages)\n",
               (unsigned long)difference);
        if (difference <= 256) {
            printf("try using   xl mem-set   to reduce its memory by 1 (Mby)\n"
                   "or perhaps reduce /local/domain/%d/memory/target by %lu",
               domid,
               (unsigned long)difference * 4);
        } else {
            printf("difference is >1Mby\n"
                   "ballon driver not running or guest still booting?");
        }
    } else if (pod_cache_pages > pod_entries) {
        estatus = 2;
        printf("SHOULD NOT HAPPEN!!! cache=%lu > outstanding=%lu",
               (unsigned long)pod_cache_pages, (unsigned long)pod_entries);
    } else if (!pod_cache_pages) {
        estatus = 0;
        printf("NOT vulnerable (not using PoD (any more))");
    } else {
        estatus = 0;
        printf("NOT vulnerable");
    }
    printf("\n");

    if (ferror(stdout) || fclose(stdout)) {
        perror("stdout");
        exit(-1);
    }
    exit(estatus);
}

From 27593ec62bdad8621df910931349d964a6dbaa8c Mon Sep 17 00:00:00 2001
From: Ian Jackson <[email protected]>
Date: Wed, 21 Oct 2015 16:18:30 +0100
Subject: [PATCH XSA-153 v3] libxl: adjust PoD target by memory fudge, too

PoD guests need to balloon at least as far as required by PoD, or risk
crashing.  Currently they don't necessarily know what the right value
is, because our memory accounting is (at the very least) confusing.

Apply the memory limit fudge factor to the in-hypervisor PoD memory
target, too.  This will increase the size of the guest's PoD cache by
the fudge factor LIBXL_MAXMEM_CONSTANT (currently 1Mby).  This ensures
that even with a slightly-off balloon driver, the guest will be
stable even under memory pressure.

There are two call sites of xc_domain_set_pod_target that need fixing:

The one in libxl_set_memory_target is straightforward.

The one in xc_hvm_build_x86.c:setup_guest is more awkward.  Simply
setting the PoD target differently does not work because the various
amounts of memory during domain construction no longer match up.
Instead, we adjust the guest memory target in xenstore (but only for
PoD guests).

This introduces a 1Mby discrepancy between the balloon target of a PoD
guest at boot, and the target set by an apparently-equivalent `xl
mem-set' (or similar) later.  This approach is low-risk for a security
fix but we need to fix this up properly in xen.git#staging and
probably also in stable trees.

This is XSA-153.

Signed-off-by: Ian Jackson <[email protected]>
---
 tools/libxl/libxl.c     |    2 +-
 tools/libxl/libxl_dom.c |    9 ++++++++-
 2 files changed, 9 insertions(+), 2 deletions(-)

diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c
index d38d0c7..1366177 100644
--- a/tools/libxl/libxl.c
+++ b/tools/libxl/libxl.c
@@ -4815,7 +4815,7 @@ retry_transaction:
     }
 
     rc = xc_domain_set_pod_target(ctx->xch, domid,
-            new_target_memkb / 4, NULL, NULL, NULL);
+            (new_target_memkb + LIBXL_MAXMEM_CONSTANT) / 4, NULL, NULL, NULL);
     if (rc != 0) {
         LIBXL__LOG_ERRNO(ctx, LIBXL__LOG_ERROR,
                 "xc_domain_set_pod_target domid=%d, memkb=%d "
diff --git a/tools/libxl/libxl_dom.c b/tools/libxl/libxl_dom.c
index b514377..8019f4e 100644
--- a/tools/libxl/libxl_dom.c
+++ b/tools/libxl/libxl_dom.c
@@ -486,6 +486,7 @@ int libxl__build_post(libxl__gc *gc, uint32_t domid,
     xs_transaction_t t;
     char **ents;
     int i, rc;
+    int64_t mem_target_fudge;
 
     if (info->num_vnuma_nodes && !info->num_vcpu_soft_affinity) {
         rc = set_vnuma_affinity(gc, domid, info);
@@ -518,11 +519,17 @@ int libxl__build_post(libxl__gc *gc, uint32_t domid,
         }
     }
 
+    mem_target_fudge =
+        (info->type == LIBXL_DOMAIN_TYPE_HVM &&
+         info->max_memkb > info->target_memkb)
+        ? LIBXL_MAXMEM_CONSTANT : 0;
+
     ents = libxl__calloc(gc, 12 + (info->max_vcpus * 2) + 2, sizeof(char *));
     ents[0] = "memory/static-max";
     ents[1] = GCSPRINTF("%"PRId64, info->max_memkb);
     ents[2] = "memory/target";
-    ents[3] = GCSPRINTF("%"PRId64, info->target_memkb - info->video_memkb);
+    ents[3] = GCSPRINTF("%"PRId64, info->target_memkb - info->video_memkb
+                        - mem_target_fudge);
     ents[4] = "memory/videoram";
     ents[5] = GCSPRINTF("%"PRId64, info->video_memkb);
     ents[6] = "domid";
-- 
1.7.10.4