ixgbe: improve MDIO performance by reducing semaphore/IPC delays

Each MDIO transfer takes 8ms, and all of that is spent in a set
of DELAY() calls, which is consuming a LOT of CPU.

* Change the timeout in ixgbe_hic_unlocked() - doing IPC to the
  on-chip firmware - to a 1uS delay and bump timeout appropriately.
  This seems to finish in a couple of microseconds on my local
  test devices.

* Change the 2ms sleep in ixgbe_release_swfw_sync_X540() to 2ms
  for EEPROM/flash, and 10uS otherwise.  10uS seems to work fine
  on my local testing, but the config EEPROM doesn't read right
  without this extra delay.

The first change shouldn't change the driver behaviour, but the
latter change is more intrusive and needs some wider testing.
(My guess here is that there SHOULD have been some completion
check somewhere in the EEPROM/flash IO path, and these sleeps
are masking them.)

Locally tested:

* C3558 (Denverton) w/ X553 backplane

Reviewed by:	kbowling
Differential Revision:	https://reviews.freebsd.org/D50295

2 files changed, 17 insertions, 3 deletions

diff --git a/sys/dev/ixgbe/ixgbe_common.c b/sys/dev/ixgbe/ixgbe_common.c
index bff022585a03..9e827d2e5473 100644
--- a/sys/dev/ixgbe/ixgbe_common.c
+++ b/sys/dev/ixgbe/ixgbe_common.c
@@ -4631,11 +4631,11 @@ s32 ixgbe_hic_unlocked(struct ixgbe_hw *hw, u32 *buffer, u32 length,
 	/* Setting this bit tells the ARC that a new command is pending. */
 	IXGBE_WRITE_REG(hw, IXGBE_HICR, hicr | IXGBE_HICR_C);
 
-	for (i = 0; i < timeout; i++) {
+	for (i = 0; i < timeout * 1000; i++) {
 		hicr = IXGBE_READ_REG(hw, IXGBE_HICR);
 		if (!(hicr & IXGBE_HICR_C))
 			break;
-		msec_delay(1);
+		usec_delay(1);
 	}
 
 	/* For each command except "Apply Update" perform
diff --git a/sys/dev/ixgbe/ixgbe_x540.c b/sys/dev/ixgbe/ixgbe_x540.c
index 57cec5b52e18..561fe6f0f78a 100644
--- a/sys/dev/ixgbe/ixgbe_x540.c
+++ b/sys/dev/ixgbe/ixgbe_x540.c
@@ -878,7 +878,21 @@ void ixgbe_release_swfw_sync_X540(struct ixgbe_hw *hw, u32 mask)
 	IXGBE_WRITE_REG(hw, IXGBE_SWFW_SYNC_BY_MAC(hw), swfw_sync);
 
 	ixgbe_release_swfw_sync_semaphore(hw);
-	msec_delay(2);
+
+	/*
+	 * EEPROM / flash access requires a 2ms sleep or interacting with
+	 * them isn't stable.  However, a 2ms delay for all sync operations
+	 * is very expensive for MDIO access.
+	 *
+	 * So use a 10us delay for PHY0/PHY1 MDIO and management access and
+	 * 2ms for everything else.  This keep MDIO access (eg from a switch
+	 * driver) fast.
+	 */
+	if (mask &
+	    (IXGBE_GSSR_PHY0_SM | IXGBE_GSSR_PHY1_SM | IXGBE_GSSR_SW_MNG_SM))
+		usec_delay(10);
+	else
+		usec_delay(2000);
 }
 
 /**