x86/xen: fix migration when ACPI suspend is not available

Xen PVH guests expose a very minimal set of ACPI tables, and due to the lack of
SCI interrupt FreeBSD doesn't allocate the suspend stacks for saving CPU and
FPU contexts.

Lack of allocated stacks would lead to a page-fault in cpususpend_handler() when
CPUs attempted to use the save context area as a result of a Xen suspend
request.  However there's no need to save the CPU or the FPU registers in the
Xen case, as that's all handled by the hypervisor.  Hence avoid saving all this
state if the suspend stacks are not allocated.

Note that this will currently only apply to PVH guests, HVM ones will still get
the stack allocated and the context saved even when not strictly required.  I
find it easier rather that having to provide cpususpend_handler() with extra
information whether the context needs to be saved or not.

Sponsored by: Cloud Software Group
Reviewed by: markj
Differential revision: https://reviews.freebsd.org/D43765

1 files changed, 18 insertions, 0 deletions

diff --git a/sys/x86/x86/mp_x86.c b/sys/x86/x86/mp_x86.c
index 11b11471d736..1027c2c8972b 100644
--- a/sys/x86/x86/mp_x86.c
+++ b/sys/x86/x86/mp_x86.c
@@ -1594,6 +1594,24 @@ cpususpend_handler(void)
 	mtx_assert(&smp_ipi_mtx, MA_NOTOWNED);
 
 	cpu = PCPU_GET(cpuid);
+
+#ifdef XENHVM
+	/*
+	 * Some Xen guest types (PVH) expose a very minimal set of ACPI tables,
+	 * and for example have no support for SCI.  That leads to the suspend
+	 * stacks not being allocated, and hence when attempting to perform a
+	 * Xen triggered suspension FreeBSD will hit a #PF.  Avoid saving the
+	 * CPU and FPU contexts if the stacks are not allocated, as the
+	 * hypervisor will already take care of this.  Note that we could even
+	 * do this for Xen triggered suspensions on guests that have full ACPI
+	 * support, but doing so would introduce extra complexity.
+	 */
+	if (susppcbs == NULL) {
+		KASSERT(vm_guest == VM_GUEST_XEN, ("Missing suspend stack"));
+		CPU_SET_ATOMIC(cpu, &suspended_cpus);
+		CPU_SET_ATOMIC(cpu, &resuming_cpus);
+	} else
+#endif
 	if (savectx(&susppcbs[cpu]->sp_pcb)) {
 #ifdef __amd64__
 		fpususpend(susppcbs[cpu]->sp_fpususpend);