Add an initial NUMA affinity/policy configuration for threads and processes.

This is based on work done by jeff@ and jhb@, as well as the numa.diff
patch that has been circulating when someone asks for first-touch NUMA
on -10 or -11.

* Introduce a simple set of VM policy and iterator types.
* tie the policy types into the vm_phys path for now, mirroring how
  the initial first-touch allocation work was enabled.
* add syscalls to control changing thread and process defaults.
* add a global NUMA VM domain policy.
* implement a simple cascade policy order - if a thread policy exists, use it;
  if a process policy exists, use it; use the default policy.
* processes inherit policies from their parent processes, threads inherit
  policies from their parent threads.
* add a simple tool (numactl) to query and modify default thread/process
  policities.
* add documentation for the new syscalls, for numa and for numactl.
* re-enable first touch NUMA again by default, as now policies can be
  set in a variety of methods.

This is only relevant for very specific workloads.

This doesn't pretend to be a final NUMA solution.

The previous defaults in -HEAD (with MAXMEMDOM set) can be achieved by
'sysctl vm.default_policy=rr'.

This is only relevant if MAXMEMDOM is set to something other than 1.
Ie, if you're using GENERIC or a modified kernel with non-NUMA, then
this is a glorified no-op for you.

Thank you to Norse Corp for giving me access to rather large
(for FreeBSD!) NUMA machines in order to develop and verify this.

Thank you to Dell for providing me with dual socket sandybridge
and westmere v3 hardware to do NUMA development with.

Thank you to Scott Long at Netflix for providing me with access
to the two-socket, four-domain haswell v3 hardware.

Thank you to Peter Holm for running the stress testing suite
against the NUMA branch during various stages of development!

Tested:

* MIPS (regression testing; non-NUMA)
* i386 (regression testing; non-NUMA GENERIC)
* amd64 (regression testing; non-NUMA GENERIC)
* westmere, 2 socket (thankyou norse!)
* sandy bridge, 2 socket (thankyou dell!)
* ivy bridge, 2 socket (thankyou norse!)
* westmere-EX, 4 socket / 1TB RAM (thankyou norse!)
* haswell, 2 socket (thankyou norse!)
* haswell v3, 2 socket (thankyou dell)
* haswell v3, 2x18 core (thankyou scott long / netflix!)

* Peter Holm ran a stress test suite on this work and found one
  issue, but has not been able to verify it (it doesn't look NUMA
  related, and he only saw it once over many testing runs.)

* I've tested bhyve instances running in fixed NUMA domains and cpusets;
  all seems to work correctly.

Verified:

* intel-pcm - pcm-numa.x and pcm-memory.x, whilst selecting different
  NUMA policies for processes under test.

Review:

This was reviewed through phabricator (https://reviews.freebsd.org/D2559)
as well as privately and via emails to freebsd-arch@.  The git history
with specific attributes is available at https://github.com/erikarn/freebsd/
in the NUMA branch (https://github.com/erikarn/freebsd/compare/local/adrian_numa_policy).

This has been reviewed by a number of people (stas, rpaulo, kib, ngie,
wblock) but not achieved a clear consensus.  My hope is that with further
exposure and testing more functionality can be implemented and evaluated.

Notes:

* The VM doesn't handle unbalanced domains very well, and if you have an overly
  unbalanced memory setup whilst under high memory pressure, VM page allocation
  may fail leading to a kernel panic.  This was a problem in the past, but it's
  much more easily triggered now with these tools.

* This work only controls the path through vm_phys; it doesn't yet strongly/predictably
  affect contigmalloc, KVA placement, UMA, etc.  So, driver placement of memory
  isn't really guaranteed in any way.  That's next on my plate.

Sponsored by:	Norse Corp, Inc.; Dell

3 files changed, 421 insertions, 0 deletions

diff --git a/usr.bin/numactl/Makefile b/usr.bin/numactl/Makefile
new file mode 100644
index 000000000000..715848724cf7
--- /dev/null
+++ b/usr.bin/numactl/Makefile
@@ -0,0 +1,5 @@
+# $FreeBSD$
+
+PROG=   numactl
+
+.include <bsd.prog.mk>
diff --git a/usr.bin/numactl/numactl.1 b/usr.bin/numactl/numactl.1
new file mode 100644
index 000000000000..750e23b5125b
--- /dev/null
+++ b/usr.bin/numactl/numactl.1
@@ -0,0 +1,132 @@
+.\" Copyright (c) 2015 Adrian Chadd <adrian@FreeBSD.org>
+.\" All rights reserved.
+.\"
+.\" Redistribution and use in source and binary forms, with or without
+.\" modification, are permitted provided that the following conditions
+.\" are met:
+.\" 1. Redistributions of source code must retain the above copyright
+.\"    notice, this list of conditions and the following disclaimer.
+.\" 2. Redistributions in binary form must reproduce the above copyright
+.\"    notice, this list of conditions and the following disclaimer in the
+.\"    documentation and/or other materials provided with the distribution.
+.\"
+.\" THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+.\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+.\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+.\" ARE DISCLAIMED.  IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
+.\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+.\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+.\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+.\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+.\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+.\" SUCH DAMAGE.
+.\"
+.\" $FreeBSD$
+.\"
+.Dd May 9, 2015
+.Dt NUMACTL 1
+.Os
+.Sh NAME
+.Nm numactl
+.Nd "manage NUMA policy configuration"
+.Sh SYNOPSIS
+.Nm
+.Op Fl l Ar policy
+.Op Fl m Ar domain
+.Op Fl c Ar domain
+.Ar cmd ...
+.Nm
+.Fl g
+.Op Fl p Ar pid
+.Op Fl t Ar tid
+.Nm
+.Fl s
+.Op Fl l Ar policy
+.Op Fl m Ar domain
+.Op Fl c Ar domain
+.Op Fl p Ar pid
+.Op Fl t Ar tid
+.Sh DESCRIPTION
+The
+.Nm
+command can be used to assign NUMA policies to processes/threads,
+run commands with a given NUMA policy, and query information
+about NUMA policies on running processes.
+.Pp
+.Nm
+requires a target to modify or query.
+The target may be specified as a command, process id or a thread id.
+Using
+.Fl -get
+the target's NUMA policy may be queried.
+Using
+.Fl -set
+the target's NUMA policy may be queried.
+If no target is specified,
+.Nm
+operates on itself.
+Not all combinations of operations and targets are supported.
+For example,
+you may not set the id of an existing set or query and launch a command
+at the same time.
+.Pp
+Each process and thread has a NUMA policy.
+By default the policy is NONE.
+If a thread policy is NONE, then the policy will fall back to the process.
+If the process policy is NONE, then the policy will fall back to the
+system default.
+The policy may be queried by using
+.Fl -get.
+.Pp
+The options are as follows:
+.Bl -tag -width ".Fl -cpudomain Ar domain"
+.It Fl -cpudomain Ar domain , Fl c Ar domain
+Set the given CPU scheduling policy.
+Constrain the the object (tid, pid, command) to run on CPUs
+that belong to the given domain.
+.It Fl -get , Fl g
+Retrieve the NUMA policy for the given thread or process id.
+.It Fl -set , Fl s
+Set the NUMA policy for the given thread or process id.
+.It Fl -memdomain Ar domain , Fl m Ar domain
+Constrain the object (tid, pid, command) to the given
+domain.
+This is only valid for fixed-domain and fixed-domain-rr.
+It must not be set for other policies.
+.It Fl -mempolicy Ar policy , Fl l Ar policy
+Set the given memory allocation policy.
+Valid policies are none, rr, fixed-domain, fixed-domain-rr,
+first-touch, and first-touch-rr.
+A memdomain argument is required for fixed-domain and
+fixed-domain-rr.
+.It Fl -pid Ar pid , Fl p Ar pid
+Operate on the given pid.
+.It Fl -tid Ar tid , Fl t Ar tid
+Operate on the given tid.
+.El
+.Sh EXIT STATUS
+.Ex -std
+.Sh EXAMPLES
+Create a
+.Pa /bin/sh
+process with memory coming from domain 0, but
+CPUs coming from domain 1:
+.Dl numactl --mempolicy=fixed-domain --memdomain=0 --cpudomain=1 /bin/sh
+.Pp
+Query the NUMA policy for the
+.Aq sh pid :
+.Dl numactl --get --pid=<sh pid>
+.Pp
+Set the NUMA policy for the given TID to round-robin:
+.Dl numactl --set --mempolicy=rr --tid=<tid>
+.Sh SEE ALSO
+.Xr cpuset 2 ,
+.Xr numa 4
+.Sh HISTORY
+The
+.Nm
+command first appeared in
+.Fx 11.0 .
+.Sh AUTHORS
+.An Adrian Chadd Aq Mt adrian@FreeBSD.org
diff --git a/usr.bin/numactl/numactl.c b/usr.bin/numactl/numactl.c
new file mode 100644
index 000000000000..ce1dfaecabc7
--- /dev/null
+++ b/usr.bin/numactl/numactl.c
@@ -0,0 +1,284 @@
+/*
+ * Copyright (c) 2015 Adrian Chadd <adrian@FreeBSD.org>
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED.  IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
+ * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+ * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+#include <sys/cdefs.h>
+__FBSDID("$FreeBSD$");
+
+#include <sys/param.h>
+#include <sys/cpuset.h>
+#include <sys/numa.h>
+#include <sys/resource.h>
+#include <sys/time.h>
+
+#include <ctype.h>
+#include <err.h>
+#include <errno.h>
+#include <getopt.h>
+#include <libgen.h>
+#include <limits.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sysexits.h>
+#include <unistd.h>
+
+static struct option longopts[] = {
+	{ "tid",	required_argument,	NULL,	't' },
+	{ "pid",	required_argument,	NULL,	'p' },
+	{ "memdomain",	required_argument,	NULL,	'm' },
+	{ "cpudomain",	required_argument,	NULL,	'c' },
+	{ "mempolicy",	required_argument, 	NULL,	'l' },
+	{ "set",	no_argument,	NULL,	's' },
+	{ "get",	no_argument,	NULL,	'g' },
+	{ NULL, 0, NULL, 0 }
+};
+
+static const char *
+policy_to_str(vm_domain_policy_type_t vt)
+{
+
+	switch (vt) {
+	case VM_POLICY_NONE:
+		return ("none");
+	case VM_POLICY_ROUND_ROBIN:
+		return ("rr");
+	case VM_POLICY_FIXED_DOMAIN:
+		return ("fixed-domain");
+	case VM_POLICY_FIXED_DOMAIN_ROUND_ROBIN:
+		return ("fixed-domain-rr");
+	case VM_POLICY_FIRST_TOUCH:
+		return ("first-touch");
+	case VM_POLICY_FIRST_TOUCH_ROUND_ROBIN:
+		return ("first-touch-rr");
+	default:
+		return ("unknown");
+	}
+}
+
+static int
+parse_policy(struct vm_domain_policy_entry *vd, const char *str)
+{
+
+	if (strcmp(str, "rr") == 0) {
+		vd->policy = VM_POLICY_ROUND_ROBIN;
+		vd->domain = -1;
+		return (0);
+	}
+
+	if (strcmp(str, "first-touch-rr") == 0) {
+		vd->policy = VM_POLICY_FIRST_TOUCH_ROUND_ROBIN;
+		vd->domain = -1;
+		return (0);
+	}
+
+	if (strcmp(str, "first-touch") == 0) {
+		vd->policy = VM_POLICY_FIRST_TOUCH;
+		vd->domain = -1;
+		return (0);
+	}
+
+	if (strcmp(str, "fixed-domain") == 0) {
+		vd->policy = VM_POLICY_FIXED_DOMAIN;
+		vd->domain = 0;
+		return (0);
+	}
+
+	if (strcmp(str, "fixed-domain-rr") == 0) {
+		vd->policy = VM_POLICY_FIXED_DOMAIN_ROUND_ROBIN;
+		vd->domain = 0;
+		return (0);
+	}
+
+	return (-1);
+}
+
+static void
+usage(void)
+{
+
+	printf("usage: numactl --get [--tid/-t <tid>] [--pid/-p <pid>]\n");
+	printf("       numactl --set [--tid=<tid>] [--pid/-p<pid>]\n");
+	printf("                     [--mempolicy/-l <policy>] [--memdomain/"
+	    "-m <domain>]\n");
+	printf("                     [--cpudomain/-c <domain>]\n");
+	printf("       numactl [--mempolicy/-l <policy>] [--memdomain/-m "
+	    "<domain>]\n");
+	printf("               [--cpudomain/-c <domain>] <cmd> ...\n");
+
+	exit(EX_USAGE);
+}
+
+static int
+set_numa_domain_cpuaffinity(int cpu_domain)
+{
+	cpuset_t set;
+	int error;
+
+	error = cpuset_getaffinity(CPU_LEVEL_WHICH, CPU_WHICH_DOMAIN,
+	    cpu_domain, sizeof(set), &set);
+	if (error != 0)
+		err(1, "cpuset_getaffinity");
+	error = cpuset_setaffinity(CPU_LEVEL_WHICH, CPU_WHICH_PID, -1,
+	    sizeof(set), &set);
+	if (error != 0)
+		err(1, "cpuset_setaffinity");
+
+	return (0);
+}
+
+int
+main(int argc, char *argv[])
+{
+	struct vm_domain_policy_entry vd;
+	lwpid_t tid;
+	pid_t pid;
+	cpuwhich_t which;
+	id_t id;
+	int error;
+	int is_set, is_get;
+	int mem_policy_set;
+	int ch;
+	int cpu_domain;
+
+	id = -1;
+	which = -1;
+	is_set = 0;
+	is_get = 0;
+	mem_policy_set = 0;
+	tid = -1;
+	pid = -1;
+	cpu_domain = -1;
+
+	while ((ch = getopt_long(argc, argv, "c:gl:m:p:st:", longopts,
+	    NULL)) != -1) {
+		switch (ch) {
+		case 'c':
+			cpu_domain = atoi(optarg);
+			break;
+		case 'g':
+			is_get = 1;
+			break;
+		case 'l':
+			if (parse_policy(&vd, optarg) != 0) {
+				fprintf(stderr,
+				    "Could not parse policy: '%s'\n", optarg);
+				exit(1);
+			}
+			mem_policy_set = 1;
+			break;
+		case 'm':
+			if (mem_policy_set == 0) {
+				fprintf(stderr,
+				    "Error: set policy first before domain\n");
+				exit(1);
+			}
+			vd.domain = atoi(optarg);
+			break;
+		case 'p':
+			pid = atoi(optarg);
+			break;
+		case 's':
+			is_set = 1;
+			break;
+		case 't':
+			tid = atoi(optarg);
+			break;
+		default:
+			usage();
+		}
+	}
+	argc -= optind;
+	argv += optind;
+
+	/* Handle the user wishing to run a command */
+	if (argc) {
+		/* Ensure that a policy was set */
+		if (mem_policy_set == 0) {
+			fprintf(stderr, "Error: no policy given\n");
+			usage();
+		}
+
+		/* Set current memory process policy, will be inherited */
+		if (numa_setaffinity(CPU_WHICH_PID, -1, &vd) != 0)
+			err(1, "numa_setaffinity");
+
+		/* If a CPU domain policy was given, include that too */
+		if (cpu_domain != -1)
+			(void) set_numa_domain_cpuaffinity(cpu_domain);
+
+		errno = 0;
+		execvp(*argv, argv);
+		err(errno == ENOENT ? 127 : 126, "%s", *argv);
+	}
+
+	/* Figure out which */
+	if (tid != -1) {
+		which = CPU_WHICH_TID;
+		id = tid;
+	} else if (pid != -1) {
+		which = CPU_WHICH_PID;
+		id = pid;
+	} else {
+		fprintf(stderr, "Error: one of tid or pid must be given\n");
+		usage();
+	}
+
+	/* Sanity checks */
+	if (is_set && is_get) {
+		fprintf(stderr, "Error: can't set both 'set' and 'get'\n");
+		usage();
+	}
+
+	if (is_set && ! mem_policy_set) {
+		fprintf(stderr, "Error: --set given, but no policy\n");
+		usage();
+	}
+
+	/* If it's get, then get the policy and return */
+	if (is_get) {
+		error = numa_getaffinity(which, id, &vd);
+		if (error != 0)
+			err(1, "numa_getaffinity");
+		printf("  Policy: %s; domain: %d\n",
+		    policy_to_str(vd.policy),
+		    vd.domain);
+		exit(0);
+	}
+
+	/* Assume it's set */
+
+	/* Syscall */
+	error = numa_setaffinity(which, id, &vd);
+	if (error != 0)
+		err(1, "numa_setaffinity");
+
+	/* If a CPU domain policy was given, include that too */
+	if (cpu_domain != -1)
+		(void) set_numa_domain_cpuaffinity(cpu_domain);
+
+	exit(0);
+}