ice: Honor iflib transmit completion batching

ice marked every transmitted packet RS and recorded every last
descriptor in its report-status queue.  Hardware therefore wrote
descriptor status for every packet, and the driver traversed every
packet while reclaiming completed descriptors.

iflib marks selected packets with IPI_TX_INTR as completion
checkpoints.  It forces a checkpoint as deferred work or ring pressure
grows.  Retain EOP on every packet, but set RS and record the descriptor
only at those checkpoints.

DPDK uses the same sparse-RS design and defaults tx_rs_thresh to 32.
Let iflib choose the adaptive interval for FreeBSD.  This reduces PCIe
and memory traffic while preserving bounded descriptor reclamation.

Validated on an E810-XXV in an A-B-A test with five matched
four-stream, TSO-disabled transmit runs per phase.  Median throughput
was 9.413, 9.413, and 9.414 Gbps.  Median whole-system CPU was 21.54%,
17.75%, and 21.62%, respectively.  The candidate used less system CPU
than both exact-baseline phases in every pair.  Interrupt rate was
unchanged, no transmit watchdog fired, and normal TSO traffic remained
line-rate.  DTrace confirmed that only iflib-selected packet-final
descriptors carried RS after the change.  The effect should be more
profound at 200Gbps but my DUT is network limited.

Reviewed by:	gallatin
MFC after:	2 weeks
Sponsored by:	BBOX.io
Differential Revision:	https://reviews.freebsd.org/D58910

1 files changed, 12 insertions, 7 deletions

diff --git a/sys/dev/ice/ice_iflib_txrx.c b/sys/dev/ice/ice_iflib_txrx.c
index a090717e7c5e..1c89b85ef4b1 100644
--- a/sys/dev/ice/ice_iflib_txrx.c
+++ b/sys/dev/ice/ice_iflib_txrx.c
@@ -138,10 +138,14 @@ _ice_ift_txd_encap(struct ice_tx_queue *txq, if_pkt_info_t pi)
 	bus_dma_segment_t *segs = pi->ipi_segs;
 	struct ice_tx_desc *txd = NULL;
 	int i, j, mask, pidx_last;
-	u32 cmd, off;
+	u32 cmd, off, tx_intr, txd_cmd;
 
 	cmd = off = 0;
 	i = pi->ipi_pidx;
+	tx_intr = pi->ipi_flags & IPI_TX_INTR;
+	txd_cmd = ICE_TX_DESC_CMD_EOP;
+	if (tx_intr)
+		txd_cmd |= ICE_TX_DESC_CMD_RS;
 
 	/* Set up the TSO/CSUM offload */
 	if (pi->ipi_csum_flags & ICE_CSUM_OFFLOAD) {
@@ -177,14 +181,15 @@ _ice_ift_txd_encap(struct ice_tx_queue *txq, if_pkt_info_t pi)
 	}
 
 	/* Set the last descriptor for report */
-#define ICE_TXD_CMD (ICE_TX_DESC_CMD_EOP | ICE_TX_DESC_CMD_RS)
 	txd->cmd_type_offset_bsz |=
-	    htole64(((u64)ICE_TXD_CMD << ICE_TXD_QW1_CMD_S));
+	    htole64((u64)txd_cmd << ICE_TXD_QW1_CMD_S);
 
-	/* Add to report status array */
-	txq->tx_rsq[txq->tx_rs_pidx] = pidx_last;
-	txq->tx_rs_pidx = (txq->tx_rs_pidx+1) & mask;
-	MPASS(txq->tx_rs_pidx != txq->tx_rs_cidx);
+	/* Add to report status array if requesting descriptor writeback. */
+	if (tx_intr) {
+		txq->tx_rsq[txq->tx_rs_pidx] = pidx_last;
+		txq->tx_rs_pidx = (txq->tx_rs_pidx+1) & mask;
+		MPASS(txq->tx_rs_pidx != txq->tx_rs_cidx);
+	}
 
 	pi->ipi_new_pidx = i;