iflib: handle transient errors from isc_txd_encap()

Until we introduced support for nic ktls offload, all error returns
from isc_txd_encap() indicated a permanent failure. Iflib remapped all
those failures to ENOMEM, which was treated by the tcp stack as a
permanent error and passed back to the caller. This was done to avoid
creating "infinite loops" where a packet couldn't be mapped for
transmit, and kept being sent over and over.

Now that we have support for nic ktls offload, some ktls offload
drivers may return ENOBUFS from their encap function to indicate that,
for example, the ktls context may not yet be fully initialized.  This
needs to be treated as a transient error so that the TCP stack may
re-try at a later time. To achieve this, pass the raw error back to
the caller when the encap routine returns an error aside from EFBIG.

Note that I audited all in-tree iflib drivers. Only ice and ixl ran
return anything other than 0 from their encap, which is EFBIG. both of
which are still treated as they were before.

Testing with an out-of-tree ktls offload nic using iflib showed ENOMEM
errors from sendmsg() before this change, and no more errors passed
back to userspace after this change.

Reviewed by: kp, sumit.saxena_broadcom.com
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D57550

1 files changed, 5 insertions, 2 deletions

diff --git a/sys/net/iflib.c b/sys/net/iflib.c
index 1af9d22db9a2..918dddc51a4f 100644
--- a/sys/net/iflib.c
+++ b/sys/net/iflib.c
@@ -3687,8 +3687,9 @@ defrag:
 				remap = 1;
 				goto defrag;
 			}
+			goto defrag_failed;
 		}
-		goto defrag_failed;
+		goto out_with_error;
 	}
 	/*
 	 * err can't possibly be non-zero here, so we don't neet to test it
@@ -3697,13 +3698,15 @@ defrag:
 	return (err);
 
 defrag_failed:
+	err = ENOMEM;
 	txq->ift_mbuf_defrag_failed++;
+out_with_error:
 	txq->ift_map_failed++;
 	m_freem(*m_headp);
 	DBG_COUNTER_INC(tx_frees);
 	*m_headp = NULL;
 	DBG_COUNTER_INC(encap_txd_encap_fail);
-	return (ENOMEM);
+	return (err);
 }
 
 static void