arm64: close a race in SVE register management

While testing an unrelated pmap change, D58708, that dramatically
reduces the number of TLBI instructions performed, and likely the
timing of unrelated events, I started seeing "Storing an invalid VFP
state" panics in vfp_save_state_common().  However, the origin of this
panic is elsewhere, in the else branch of sve_restore_state().
Specifically, my pmap change seems to have increased the likelihood that
the thread executing the else branch would be preempted by another
thread between the critical_exit() inside the else branch's call to
vfp_restore_state_common() and its own call to critical_enter().  Prior
to expanding the scope of the else branch's critical section, the MPASS
added by this change would fire, catching the problem at its source,
rather than later in vfp_save_state_common().

Assisted-by:	Claude Code (Opus 5)
Reviewed by:	kib, markj
MFC after: 	1 week
Differential Revision:	https://reviews.freebsd.org/D58723

1 files changed, 4 insertions, 2 deletions

diff --git a/sys/arm64/arm64/vfp.c b/sys/arm64/arm64/vfp.c
index d4c0eacf690c..11faa4cc21cd 100644
--- a/sys/arm64/arm64/vfp.c
+++ b/sys/arm64/arm64/vfp.c
@@ -797,15 +797,17 @@ sve_restore_state(struct thread *td)
 
 		critical_exit();
 	} else {
+		critical_enter();
+
 		vfp_restore_state_common(td, curpcb->pcb_fpflags);
 
 		/* Enable SVE if it wasn't previously enabled */
 		if ((curpcb->pcb_fpflags & PCB_FP_SVEVALID) == 0) {
-			critical_enter();
+			MPASS(PCPU_GET(fpcurthread) == td);
 			sve_enable();
 			curpcb->pcb_fpflags |= PCB_FP_SVEVALID;
-			critical_exit();
 		}
+		critical_exit();
 	}
 
 	return (true);