PCI hot-plug: use dedicated taskqueue for device attach / detach

Attaching and detaching devices can be heavy-weight and detaching can
sleep waiting for events.  For that reason using the system-wide
single-threaded taskqueue_thread is not really appropriate.
There is even a possibility for a deadlock if taskqueue_thread is used
for detaching.

In fact, there is an easy to reproduce deadlock involving nvme, pass
and a sudden removal of an NVMe device.
A pass peripheral would not release a reference on an nvme sim until
pass_shutdown_kqueue() is executed via taskqueue_thread.  But the
taskqueue's thread is blocked in nvme_detach() -> ... -> cam_sim_free()
because of the outstanding reference.

Sponsored by:	CyberSecure
Reviewed by:	mav, imp

(cherry picked from commit 12588ce02dd835b332952d9fece5881d943554a9)

1 files changed, 4 insertions, 2 deletions

diff --git a/sys/dev/pci/pci_pci.c b/sys/dev/pci/pci_pci.c
index ff3dcdc84c47..4b7560f499ef 100644
--- a/sys/dev/pci/pci_pci.c
+++ b/sys/dev/pci/pci_pci.c
@@ -922,6 +922,8 @@ SYSCTL_INT(_hw_pci, OID_AUTO, enable_pcie_hp, CTLFLAG_RDTUN,
     &pci_enable_pcie_hp, 0,
     "Enable support for native PCI-express HotPlug.");
 
+TASKQUEUE_DEFINE_THREAD(pci_hp);
+
 static void
 pcib_probe_hotplug(struct pcib_softc *sc)
 {
@@ -1151,7 +1153,7 @@ pcib_pcie_hotplug_update(struct pcib_softc *sc, uint16_t val, uint16_t mask,
 	 */
 	if (schedule_task &&
 	    (pcib_hotplug_present(sc) != 0) != (sc->child != NULL))
-		taskqueue_enqueue(taskqueue_thread, &sc->pcie_hp_task);
+		taskqueue_enqueue(taskqueue_pci_hp, &sc->pcie_hp_task);
 }
 
 static void
@@ -1443,7 +1445,7 @@ pcib_detach_hotplug(struct pcib_softc *sc)
 	error = pcib_release_pcie_irq(sc);
 	if (error)
 		return (error);
-	taskqueue_drain(taskqueue_thread, &sc->pcie_hp_task);
+	taskqueue_drain(taskqueue_pci_hp, &sc->pcie_hp_task);
 	callout_drain(&sc->pcie_ab_timer);
 	callout_drain(&sc->pcie_cc_timer);
 	callout_drain(&sc->pcie_dll_timer);