No functional change.

MFC after:	1 week
Sponsored by:	Chelsio Communications

This allows iw_cxgbe.ko, cxgbei.ko, nvmf_che.ko, etc. to be loaded when
debug.link_elf_leak_locals and debug.link_elf_obj_leak_locals are
disabled.

PR:		291250
MFC after:	1 week
Sponsored by:	Chelsio Communications

Fixes: ec0cd28

This provides an alternative NVMe over TCP transport which uses PDU
offload for TOE connections on a T7.

Similar to iSCSI offload via cxgbei.ko, nvmf_che uses DDP when
possible to enable the NIC to DMA received data directly into I/O data
buffers (pages from a struct bio on the host side, pages from a CTL
I/O requests on the controller side) to avoid copying data on the host
CPU.  nvmf_che is also able to receive a stream of C2H or H2C PDUs for
a single data transfer when using DDP without processing the header of
each PDU.

Unlike cxgbei, nvmf_che aims to be mostly transparent to end users.
Notably, neither nvmecontrol or ctld have to be explicitly asked to
use an offload.  Instead, TCP queue pairs are claimed by this driver
whenever they are eligible (e.g., using TOE).

The main restriction of nvmf_che compared to the software TCP
transport is that Chelsio adapters have a restriction on the largest
PDU that can be sent and received.  When sending data, nvmf_che is
able to split large C2H or H2C data requests across multiple PDUs
without affecting nvmf(4) or nvmft(4).

To avoid overly large PDUs when using nvmf(4), nvmf_che reports a data
transfer limit that is honored by nvmf(4).  This ensures that the
remote controller's PDUs will never be too large (since the command
transfer size is limited to one PDU) and also ensures that nvmf(4)
will never to try to send a command PDU with ICD that is too large.

For nvmft(4), overly large command PDUs due to ICD are avoided by
clamping the size of the reported IOCCSZ in the controller data.
However, to ensure that H2C PDUs are sufficiently small, nvmf_che will
only claim queue pairs which advertised a suitable MAXH2CDATA
parameter during queue negotiation.  For ctld(8), this can be achieved
by setting the MAXH2CDATA option in a transport-group, e.g. for T7:

transport-group tg0 {
	discovery-auth-group no-authentication
	listen tcp 0.0.0.0
	listen tcp [::]
	listen discovery-tcp 0.0.0.0
	listen discovery-tcp [::]
	option MAXH2CDATA 32488
}

Sponsored by:	Chelsio Communications

Unlike NIC KTLS support on T6, T7 is able to reuse the existing TSO
functionality directly, including trimming the output of the crypto
engine before it is passed on to TSO.  This is much simpler and does
not require the use of bypass pseudo-connections in the TOE engine.
Among other things this permits arbitrary TCP options (including the
full range of possible TCP timestamp values) while also avoiding
various edge cases where parts of a requested TCP packet could not
always be transmitted (e.g. partial trailers).  This implementation
also permits NIC KTLS to be used in parallel with TOE.

This version does not yet support connections over a VF (specifically
the ktls_tunnel_packet function needs to handle the VF work request),
nor does it support VxLAN offload.

MFC after:	3 days
Sponsored by:	Chelsio Communications

Both RDMA (iw_cxgbe) and NVMe offloads use TPT table entries to map
transaction tags in incoming PDUs to buffers in host memory permitting
direct placement of received data into host memory buffers avoiding
copies (iSCSI offload uses a different scheme for mapping tags to host
memory).  Move the vmem arenas for the supporting card memory regions
from iw_cxgbe to the main driver so they can be shared with the NVMe
offload driver.  In addition, add some helper routines for
constructing work requests to update TPT table entries.

MFC after:	3 days
Sponsored by:	Chelsio Communications

This is the first of a series of commits that will add T7 support to
cxgbe.  The ASIC is gen5x16 on the PCIe side and has a 400Gbps MAC on
the Ethernet side.  NICs using the T7 will come in the following
variants:

* 1 x 400Gbps with QSFP-DD connector
* 2 x 200/100/40Gbps with QSFP56/QSFP28/QSFP+ connectors
* 4 x 50/25/10/1Gbps with SFP28/SFP+/SFP connectors

There are 8 general purpose ARM A72 cores available on select
SmartNIC/DPU boards.

Obtained from:	Chelsio Communications
MFC after:	3 days
Sponsored by:	Chelsio Communications

This is a residual of the $FreeBSD$ removal.

MFC After: 3 days (though I'll just run the command on the branches)
Sponsored by: Netflix

Version : 1.27.5.0
Date    : 10/10/2023
=====================
Fixes
-----

BASE:
- Fixed handling the Remote Fault with AN, causing the link failure.
=====================

Obtained from:	Chelsio Communications
MFC after:	2 weeks
Sponsored by:	Chelsio Communications

Remove /^\s*#[#!]?\s*\$FreeBSD\$.*$\n/