Make lock-less delayed invalidation operational very early.

Apparently, there is more code trying to call pmap_remove() early,
mostly to free preloaded memory.  Instead of moving all deallocations
to the point where a scheduler is initialized, add missed setup of
thread0 di init at hammer_time().

The code in pmap_delayed_invl_start_u() is modified to not ever take
the thread lock if the thread priority is less or equal to PVM.  Since
thread0 starts at priority 0, and then is reset to PVM at
proc0_init(), this eliminates taking the thread lock during early
boot.

While there, fix off by one in comparision of the base priority.

Reported and tested by:	bcran (previous version)
Reviewed by:	markj
Sponsored by:	The FreeBSD Foundation
MFC after:	29 days

2 files changed, 16 insertions, 8 deletions

diff --git a/sys/amd64/amd64/machdep.c b/sys/amd64/amd64/machdep.c
index 1cf09dc5cb71..03fe8a5d0964 100644
--- a/sys/amd64/amd64/machdep.c
+++ b/sys/amd64/amd64/machdep.c
@@ -1617,6 +1617,13 @@ hammer_time(u_int64_t modulep, u_int64_t physfree)
 	physfree += kstack0_sz;
 
 	/*
+	 * Initialize enough of thread0 for delayed invalidation to
+	 * work very early.  Rely on thread0.td_base_pri
+	 * zero-initialization, it is reset to PVM at proc0_init().
+	 */
+	pmap_thread_init_invl_gen(&thread0);
+
+	/*
 	 * make gdt memory segments
 	 */
 	for (x = 0; x < NGDT; x++) {
diff --git a/sys/amd64/amd64/pmap.c b/sys/amd64/amd64/pmap.c
index 7997a9f65dc7..89ba9da19d86 100644
--- a/sys/amd64/amd64/pmap.c
+++ b/sys/amd64/amd64/pmap.c
@@ -700,16 +700,17 @@ pmap_delayed_invl_start_u(void)
 	invl_gen = &td->td_md.md_invl_gen;
 	PMAP_ASSERT_NOT_IN_DI();
 	lock_delay_arg_init(&lda, &di_delay);
-	thread_lock(td);
+	invl_gen->saved_pri = 0;
 	pri = td->td_base_pri;
-	if (pri < PVM) {
-		invl_gen->saved_pri = 0;
-	} else {
-		invl_gen->saved_pri = pri;
-		sched_prio(td, PVM);
+	if (pri > PVM) {
+		thread_lock(td);
+		pri = td->td_base_pri;
+		if (pri > PVM) {
+			invl_gen->saved_pri = pri;
+			sched_prio(td, PVM);
+		}
+		thread_unlock(td);
 	}
-	thread_unlock(td);
-
 again:
 	PV_STAT(i = 0);
 	for (p = &pmap_invl_gen_head;; p = prev.next) {