pci: Do not reconcile MPS across PCI domains

A PCI function can provide a host bridge into a synthetic PCI domain.
Intel VMD does this: the host facing VMD function remains in its
original domain while the hidden Root Ports and endpoints appear in a
separate domain.  The VMD function's Device Control does not describe
an upstream link in that synthetic hierarchy.

The hierarchy wide cold pass incorrectly used the VMD function's MPS
to reprogram the hidden ports and their endpoints.  Stop both cold
reconciliation and runtime path walks at a PCI domain boundary.  The
real Root Ports within the VMD domain continue to reconcile their
endpoints normally.

Reviewed by:	imp
Tested by:	Michael Butler <imb@protected-networks.net>
Fixes:		8e9fe9996a1f ("pci: Reconcile MPS before attaching PCIe devices")
MFC after:	6 days
Sponsored by:	BBOX.io
Differential Revision:	https://reviews.freebsd.org/D58837

1 files changed, 14 insertions, 0 deletions

diff --git a/sys/dev/pci/pci.c b/sys/dev/pci/pci.c
index 83ea487bdaa5..34ee261604a2 100644
--- a/sys/dev/pci/pci.c
+++ b/sys/dev/pci/pci.c
@@ -4488,6 +4488,13 @@ pcie_path_mps(device_t dev, uint16_t *mpsp)
 		pcib = device_get_parent(bus);
 		if (pcib == NULL || !is_pci_device(pcib))
 			break;
+		/*
+		 * A PCI function may expose a host bridge for a synthetic PCI
+		 * domain.  Its Device Control belongs to the parent domain and
+		 * does not describe an upstream link in the synthetic hierarchy.
+		 */
+		if (pci_get_domain(pcib) != pci_get_domain(dev))
+			break;
 		dinfo = device_get_ivars(pcib);
 		if (dinfo->cfg.pcie.pcie_location != 0) {
 			mps = pcie_read_config(pcib, PCIER_DEVICE_CTL, 2) &
@@ -4705,6 +4712,13 @@ pcie_reconcile_link_mps(device_t bus)
 	pcib = device_get_parent(bus);
 	if (!is_pci_device(pcib))
 		return;
+	/*
+	 * A PCI function may provide a host bridge into a separate domain,
+	 * as Intel VMD does.  Do not treat the function's host-facing PCIe
+	 * Device Control as the upstream end of a link in the child domain.
+	 */
+	if (pci_get_domain(pcib) != pcib_get_domain(bus))
+		return;
 	upinfo = device_get_ivars(pcib);
 	if (upinfo->cfg.pcie.pcie_location == 0)
 		return;