Revamp the MSI/MSI-X code a bit to achieve two main goals:

- Simplify the amount of work that has be done for each architecture by
  pushing more of the truly MI code down into the PCI bus driver.
- Don't bind MSI-X indicies to IRQs so that we can allow a driver to map
  multiple MSI-X messages into a single IRQ when handling a message
  shortage.

The changes include:
- Add a new pcib_if method: PCIB_MAP_MSI() which is called by the PCI bus
  to calculate the address and data values for a given MSI/MSI-X IRQ.
  The x86 nexus drivers map this into a call to a new 'msi_map()' function
  in msi.c that does the mapping.
- Retire the pcib_if method PCIB_REMAP_MSIX() and remove the 'index'
  parameter from PCIB_ALLOC_MSIX().  MD code no longer has any knowledge
  of the MSI-X index for a given MSI-X IRQ.
- The PCI bus driver now stores more MSI-X state in a child's ivars.
  Specifically, it now stores an array of IRQs (called "message vectors" in
  the code) that have associated address and data values, and a small
  virtual version of the MSI-X table that specifies the message vector
  that a given MSI-X table entry uses.  Sparse mappings are permitted in
  the virtual table.
- The PCI bus driver now configures the MSI and MSI-X address/data
  registers directly via custom bus_setup_intr() and bus_teardown_intr()
  methods.  pci_setup_intr() invokes PCIB_MAP_MSI() to determine the
  address and data values for a given message as needed.  The MD code
  no longer has to call back down into the PCI bus code to set these
  values from the nexus' bus_setup_intr() handler.
- The PCI bus code provides a callout (pci_remap_msi_irq()) that the MD
  code can call to force the PCI bus to re-invoke PCIB_MAP_MSI() to get
  new values of the address and data fields for a given IRQ.  The x86
  MSI code uses this when an MSI IRQ is moved to a different CPU, requiring
  a new value of the 'address' field.
- The x86 MSI psuedo-driver loses a lot of code, and in fact the separate
  MSI/MSI-X pseudo-PICs are collapsed down into a single MSI PIC driver
  since the only remaining diff between the two is a substring in a
  bootverbose printf.
- The PCI bus driver will now restore MSI-X state (including programming
  entries in the MSI-X table) on device resume.
- The interface for pci_remap_msix() has changed.  Instead of accepting
  indices for the allocated vectors, it accepts a mini-virtual table
  (with a new length parameter).  This table is an array of u_ints, where
  each value specifies which allocated message vector to use for the
  corresponding MSI-X message.  A vector of 0 forces a message to not
  have an associated IRQ.  The device may choose to only use some of the
  IRQs assigned, in which case the unused IRQs must be at the "end" and
  will be released back to the system.  This allows a driver to use the
  same remap table for different shortage values.  For example, if a driver
  wants 4 messages, it can use the same remap table (which only uses the
  first two messages) for the cases when it only gets 2 or 3 messages and
  in the latter case the PCI bus will release the 3rd IRQ back to the
  system.

MFC after:	1 month

1 files changed, 12 insertions, 12 deletions

diff --git a/sys/amd64/amd64/nexus.c b/sys/amd64/amd64/nexus.c
index b15d301d2f0b..108abc117969 100644
--- a/sys/amd64/amd64/nexus.c
+++ b/sys/amd64/amd64/nexus.c
@@ -108,9 +108,9 @@ static	int nexus_get_resource(device_t, device_t, int, int, u_long *, u_long *);
 static void nexus_delete_resource(device_t, device_t, int, int);
 static	int nexus_alloc_msi(device_t pcib, device_t dev, int count, int maxcount, int *irqs);
 static	int nexus_release_msi(device_t pcib, device_t dev, int count, int *irqs);
-static	int nexus_alloc_msix(device_t pcib, device_t dev, int index, int *irq);
-static	int nexus_remap_msix(device_t pcib, device_t dev, int index, int irq);
+static	int nexus_alloc_msix(device_t pcib, device_t dev, int *irq);
 static	int nexus_release_msix(device_t pcib, device_t dev, int irq);
+static	int nexus_map_msi(device_t pcib, device_t dev, int irq, uint64_t *addr, uint32_t *data);
 
 static device_method_t nexus_methods[] = {
 	/* Device interface */
@@ -140,8 +140,8 @@ static device_method_t nexus_methods[] = {
 	DEVMETHOD(pcib_alloc_msi,	nexus_alloc_msi),
 	DEVMETHOD(pcib_release_msi,	nexus_release_msi),
 	DEVMETHOD(pcib_alloc_msix,	nexus_alloc_msix),
-	DEVMETHOD(pcib_remap_msix,	nexus_remap_msix),
 	DEVMETHOD(pcib_release_msix,	nexus_release_msix),
+	DEVMETHOD(pcib_map_msi,		nexus_map_msi),
 
 	{ 0, 0 }
 };
@@ -504,24 +504,17 @@ nexus_delete_resource(device_t dev, device_t child, int type, int rid)
 }
 
 static int
-nexus_alloc_msix(device_t pcib, device_t dev, int index, int *irq)
+nexus_alloc_msix(device_t pcib, device_t dev, int *irq)
 {
 	int error, new;
 
-	error = msix_alloc(dev, index, irq, &new);
+	error = msix_alloc(dev, irq, &new);
 	if (new)
 		rman_manage_region(&irq_rman, *irq, *irq);
 	return (error);
 }
 
 static int
-nexus_remap_msix(device_t pcib, device_t dev, int index, int irq)
-{
-
-	return (msix_remap(index, irq));
-}
-
-static int
 nexus_release_msix(device_t pcib, device_t dev, int irq)
 {
 
@@ -551,6 +544,13 @@ nexus_release_msi(device_t pcib, device_t dev, int count, int *irqs)
 	return (msi_release(irqs, count));
 }
 
+static int
+nexus_map_msi(device_t pcib, device_t dev, int irq, uint64_t *addr, uint32_t *data)
+{
+
+	return (msi_map(irq, addr, data));
+}
+
 /* Placeholder for system RAM. */
 static void
 ram_identify(driver_t *driver, device_t parent)