Add support for multithreading the inactive queue pageout within a domain.

In very high throughput workloads, the inactive scan can become overwhelmed
as you have many cores producing pages and a single core freeing.  Since
Mark's introduction of batched pagequeue operations, we can now run multiple
inactive threads working on independent batches.

To avoid confusing the pid and other control algorithms, I (Jeff) do this in
a mpi-like fan out and collect model that is driven from the primary page
daemon.  It decides whether the shortfall can be overcome with a single
thread and if not dispatches multiple threads and waits for their results.

The heuristic is based on timing the pageout activity and averaging a
pages-per-second variable which is exponentially decayed. This is visible in
sysctl and may be interesting for other purposes.

I (Jeff) have verified that this does indeed double our paging throughput
when used with two threads. With four we tend to run into other contention
problems.  For now I would like to commit this infrastructure with only a
single thread enabled.

The number of worker threads per domain can be controlled with the
'vm.pageout_threads_per_domain' tunable.

Submitted by:	jeff (earlier version)
Discussed with:	markj
Tested by:	pho
Sponsored by:	probably Netflix (based on contemporary commits)
Differential Revision:	https://reviews.freebsd.org/D21629

1 files changed, 9 insertions, 0 deletions

diff --git a/sys/vm/vm_pagequeue.h b/sys/vm/vm_pagequeue.h
index 0573457db675..0e5d1c911b8d 100644
--- a/sys/vm/vm_pagequeue.h
+++ b/sys/vm/vm_pagequeue.h
@@ -84,6 +84,7 @@ struct vm_batchqueue {
 } __aligned(CACHE_LINE_SIZE);
 
 #include <vm/uma.h>
+#include <sys/_blockcount.h>
 #include <sys/pidctrl.h>
 struct sysctl_oid;
 
@@ -254,6 +255,14 @@ struct vm_domain {
 	/* Paging control variables, used within single threaded page daemon. */
 	struct pidctrl vmd_pid;		/* Pageout controller. */
 	boolean_t vmd_oom;
+	u_int vmd_inactive_threads;
+	u_int vmd_inactive_shortage;		/* Per-thread shortage. */
+	blockcount_t vmd_inactive_running;	/* Number of inactive threads. */
+	blockcount_t vmd_inactive_starting;	/* Number of threads started. */
+	volatile u_int vmd_addl_shortage;	/* Shortage accumulator. */
+	volatile u_int vmd_inactive_freed;	/* Successful inactive frees. */
+	volatile u_int vmd_inactive_us;		/* Microseconds for above. */
+	u_int vmd_inactive_pps;		/* Exponential decay frees/second. */
 	int vmd_oom_seq;
 	int vmd_last_active_scan;
 	struct vm_page vmd_markers[PQ_COUNT]; /* (q) markers for queue scans */