aboutsummaryrefslogtreecommitdiff
path: root/sys/dev/cxgbe/tom
Commit message (Collapse)AuthorAgeFilesLines
* cxgbe tom: Use the same WRs as iSCSI to send PDUs for NVMeJohn Baldwin2026-02-251-99/+13
| | | | | | Reviewed by: np (earlier version) Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D55470
* cxgbe(4): T7 related updates to shared codeNavdeep Parhar2025-11-131-1/+1
| | | | | | | | | | - Avoid some more registers with read side-effects during regdump. - mps_tcam_size is 3x the size of T6/T5. - Update rss_rd_row to work with T7. Obtained from: Chelsio Communications MFC after: 1 week Sponsored by: Chelsio Communications
* cxgbe tom: Don't require exact match for TCP seqno for FIN for NVMe offloadJohn Baldwin2025-11-101-1/+2
| | | | | | | | Similar to iSCSI on T6+, additional DDP data might have been received without a completion posted if a connection was closed in the middle of a transfer. Sponsored by: Chelsio Communications
* cxgbe tom: Add support for sending NVMe/TCP PDUsJohn Baldwin2025-11-101-9/+335
| | | | | | | | | | These work requests are largely similar to the same work requests for iSCSI offload but use a newer tx_data work request structure. This includes initial support for ISO where a large C2H_DATA or H2C_DATA PDU is split into multiple PDUs on the wire. Sponsored by: Chelsio Communications
* cxgbe: Add ULD_NVME for NVMe/TCP offloadJohn Baldwin2025-11-101-2/+4
| | | | | | NVMe/TCP offload depends on TOE similar to iSCSI and iWARP offloads. Sponsored by: Chelsio Communications
* cxgbe TOE TLS: Support for TLS 1.3 for receiveJohn Baldwin2025-09-291-10/+41
| | | | | MFC after: 3 days Sponsored by: Chelsio Communications
* cxgbe TOE TLS: Support for TLS 1.3 for transmit.John Baldwin2025-09-292-29/+56
| | | | | MFC after: 3 days Sponsored by: Chelsio Communications
* cxgbe TOE TLS: Better support for partial records for T7.John Baldwin2025-09-291-51/+114
| | | | | | | | | | | | T7 is able to start TOE TLS receive offload more cleanly if a partial TLS record containing a full header and at least one other byte has been received. In that case it is able to request the remaining bytes for the current TLS record from the NIC. Once those bytes has been received TLS receive offload can then be received starting with the subsequent record. MFC after: 3 days Sponsored by: Chelsio Communications
* cxgbe: Move the STAG and PBL memory pool arenas to the base driverJohn Baldwin2025-09-291-0/+6
| | | | | | | | | | | | | | Both RDMA (iw_cxgbe) and NVMe offloads use TPT table entries to map transaction tags in incoming PDUs to buffers in host memory permitting direct placement of received data into host memory buffers avoiding copies (iSCSI offload uses a different scheme for mapping tags to host memory). Move the vmem arenas for the supporting card memory regions from iw_cxgbe to the main driver so they can be shared with the NVMe offload driver. In addition, add some helper routines for constructing work requests to update TPT table entries. MFC after: 3 days Sponsored by: Chelsio Communications
* cxgbe: Move send_iscsi_flowc_wr to t4_tom.koJohn Baldwin2025-09-292-0/+45
| | | | | | | | Rename the function to send_txdataplen_max_flowc_wr to match the parameter it sets. This function will be reused by other ULP modules. MFC after: 3 days Sponsored by: Chelsio Communications
* cxgbe: Refactor find_offload_adapter and move to t4_tom from cxgbeiJohn Baldwin2025-09-292-0/+49
| | | | | | | This allows it to be used for other offload drivers. MFC after: 3 days Sponsored by: Chelsio Communications
* cxgbe(4): T7 ULPTX supports larger data length with MEMIO commandsNavdeep Parhar2025-09-291-4/+16
| | | | | MFC after: 3 days Sponsored by: Chelsio Communications
* cxgbe(4): T7 TCB CPLs have queue and channel in different locationNavdeep Parhar2025-09-292-5/+21
| | | | | MFC after: 3 days Sponsored by: Chelsio Communications
* cxgbe(4): Update the filtering code for T7Navdeep Parhar2025-09-292-14/+28
| | | | | | | | There are new fields available and the width of the optional part has changed. This affects the ntuples used by TOE/filters/hashfilters. MFC after: 3 days Sponsored by: Chelsio Communications
* cxgbe(4): Updates for T7 CIM multicore operationNavdeep Parhar2025-09-294-1/+40
| | | | | | | | | | T7 has a multicore microprocessor and each core has its own queue configuration, inbound/outbound queues, and logic analyzer. A work request involving a tid can only be handled on queues where (tid & tid_qid_sel_mask) == (eq->cntxt_id & tid_qid_sel_mask). MFC after: 3 days Sponsored by: Chelsio Communications
* cxgbe(4): hw/fw headers and shared code for the Terminator 7 ASICNavdeep Parhar2025-09-291-2/+2
| | | | | | | | | | | | | | | | | | This is the first of a series of commits that will add T7 support to cxgbe. The ASIC is gen5x16 on the PCIe side and has a 400Gbps MAC on the Ethernet side. NICs using the T7 will come in the following variants: * 1 x 400Gbps with QSFP-DD connector * 2 x 200/100/40Gbps with QSFP56/QSFP28/QSFP+ connectors * 4 x 50/25/10/1Gbps with SFP28/SFP+/SFP connectors There are 8 general purpose ARM A72 cores available on select SmartNIC/DPU boards. Obtained from: Chelsio Communications MFC after: 3 days Sponsored by: Chelsio Communications
* cxgbe(4): Use hw_port and tx_chan correctly all over the driverNavdeep Parhar2025-09-292-4/+6
| | | | | | | | | | | | | | | | | | | | The port's tx channel and its hardware index are different things that happen to have the same value on NICs using T4 through T6. This is going to change so use the correct spelling all over the driver. Specifically, the firmware expects the hw port index in all of these: * PCIe channel for queues * TX interface for TX CPLs * PORTID in FW_PORT commands * PORTID in t4_alloc_vi * flowid in all the firmware flowc WRs * mbox commands that deal with the tx scheduler Also, create a port_id lookup table for hw_port, just like the one for tx_chan. Use it to lookup the port softc in the port_info handler. MFC after: 3 days Sponsored by: Chelsio Communications
* cxgbe(4): Rename lport to hw_port.Navdeep Parhar2025-09-292-2/+2
| | | | | | | No functional change intended. MFC after: 3 days Sponsored by: Chelsio Communications
* cxgbe tom: Send auxiliary TLS work requests as raw WR mbufsJohn Baldwin2025-09-113-43/+51
| | | | | | | | | | | | | | In particular, the work request to update the TCB state when switching from plain TCP to TLS is now queued on the connection's offload transmit queue rather than over the per-port control queue. This also handles an unlikely edge case that a connection does not have sufficient credits to transmit other work requests synchronously such as programming the key in on-card memory or altering TCB fields. Reviewed by: np Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D47763
* cxgbe tom: Export alloc_raw_mbuf for use in other TOE driversJohn Baldwin2025-09-112-1/+2
| | | | | | Reviewed by: np Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D47762
* cxgbe tom: Support sending "raw" WR mbufs on plain TCP and TLS socketsJohn Baldwin2025-09-113-0/+100
| | | | | | | | | | | | This is intended to support control work requests that manipulate connection state but do not transmit data. Raw WRs are transmitted immediately if possible, otherwise they are queued to an mbufq in the toe pcb until sufficient credits are available. Raw WRs take precendence over transmitting socket data. Reviewed by: np Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D47761
* cxgbe tom: Halve the size of offload transmit software descriptorsJohn Baldwin2025-09-114-2/+27
| | | | | | | | Use bitfields to pack tx_credits and plen into a single 32-bit word. Reviewed by: np Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D47759
* sendfile: retire M_BLOCKEDGleb Smirnoff2025-07-252-5/+5
| | | | | | | | | | | | | | | | | | | Follow unix(4) commit 51ac5ee0d57f and retire M_BLOCKED for TCP sockets as well. The M_BLOCKED flag was introduced back 2016 together with non- blocking sendfile(2). It marked mbufs in a sending socket buffer that could be ready to sent, but are sitting behind an M_NOTREADY mbuf(s), that blocks them. You may consider this flag as an INVARIANT flag that helped to ensure socket buffer consistency. Or maybe the socket code was so convoluted back then, that it was unclear if sbfree() may be called on an mbuf that is in the middle of the buffer, and I decided to introduce the flag to protect against that. With today state of socket buffer code it became clear that the latter cannot happen. And this commit adds an assertion proving that. Reviewed by: markj Differential Revision: https://reviews.freebsd.org/D50728
* cxgbe(4): Block most access to the hardware as soon as the adapter stops.Navdeep Parhar2025-02-251-1/+1
| | | | | | | | | | | | | | Add a new hw_all_ok() routine and use it to avoid hardware access in the public control interfaces (ifnet ioctls, ifmedia calls, etc.). Continue to use hw_off_limits() in the private ioctls/sysctls and other debug code. Retire adapter_stopped() as it's of no use by itself. This fixes problems where ifnet slow-path operations would enter a synch_op just before set_adapter_hwstatus(false) and touch the hardware when it's not safe to do so. MFC after: 1 week Sponsored by: Chelsio Communications
* cxgbe(4): Remove smt_idx from the L2 table entry structure.Navdeep Parhar2025-02-141-4/+2
| | | | | | | | An L2 table entry isn't associated with a particular SMT (Source MAC Table) entry. MFC after: 1 week Sponsored by: Chelsio Communications
* sockets: make pr_aio_queue the default methodGleb Smirnoff2025-02-101-3/+8
| | | | | | | | | | Call it directly instead of each time calling pr_aio_queue_notsupp() and then doing the actual job. The only user of non default method was cxgbe(4). It calls now into the default method in case of own method failure. This preserves existing behavior. Reviewed by: markj, jhb Differential Revision: https://reviews.freebsd.org/D48863
* cxgbe tom: Make t4_push_frames static to t4_cpl_io.cJohn Baldwin2025-01-142-2/+1
| | | | | | | This function is not used outside of this file. Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D47760
* cxgbe tom: Restore support for zerocopy TCP receive for aio_read()John Baldwin2024-12-111-10/+10
| | | | | | | | | | | | | The commit to introduce TCP_USE_DDP support had a couple of bugs that broke support for zerocopy receive via aio_read(). First, the length and offset arguments to mk_update_tcb_for_ddp() were reversed which prevented DDP from working. Second, the AIO state in the toep was initialized too late when the first aio_read() request was queued. Reported by: Harshavardhan Tanneru @ Chelsio Fixes: eba13bbc37ab cxgbe: Support TCP_USE_DDP on offloaded TOE connections MFC after: 1 week Sponsored by: Chelsio Communications
* cxgbe/t4_tom: Plug an stid leak.Navdeep Parhar2024-12-081-0/+2
| | | | | | | | | Normally the reply to destroy_server() releases the listen context but it is not called when the adapter is suspended. Release the context right away in that case. MFC after: 1 week Sponsored by: Chelsio Communications
* tcp: extend the use of the th_flags accessor functionRichard Scheffenegger2024-11-291-1/+1
| | | | | | | | | | | Formally, there are 12 bits for TCP header flags. Use the accessor functions in more (kernel) places. No functional change. Reviewed By: cc, #transport, cy, glebius, #iflib, kbowling Sponsored by: NetApp, Inc. Differential Revision: https://reviews.freebsd.org/D47063
* mbuf: Don't force all M_EXTPG mbufs to be read-onlyJohn Baldwin2024-10-311-1/+1
| | | | | | | | | | | Some M_EXTPG mbufs are read-only (e.g. those backing sendfile requests), but others are not. Add a flags argument to mb_alloc_ext_pgs that can be used to set M_RDONLY when needed rather than setting it unconditionally. Update mb_unmapped_to_ext to preserve M_RDONLY from the unmapped mbuf. Reviewed by: gallatin Differential Revision: https://reviews.freebsd.org/D46783
* cxgbe/t4_tom: Change stid allocation strategy to be more IPv6 friendly.Navdeep Parhar2024-10-172-73/+103
| | | | | | | | | | | | | | A hardware IPv6 server needs 2 consecutive stids (server tids) starting from a 2-aligned stid whereas an IPv4 server needs only 1 stid without any constraint. The allocator used to grab the first free stid(s) for both but this can fragment the stid space leaving nothing suitable for IPv6 even when lots of stids are available. Change the allocator to prefer stids for IPv4 from the ones that cannot be used for IPv6. Reviewed by: jhb MFC after: 1 week Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D47042
* cxgbe(4): Allow t4_tom to be unloaded safely.Navdeep Parhar2024-09-301-8/+23
| | | | | | | | | | | * Disable IFCAP_TOE automatically on all ifnets on all adapters during unload. This is user-friendly and avoids panics due to stale ifnet state after t4_tom is unloaded. * Do not allow unload if tids are in use by the TOE on any adapter. Reported by: Bimal Abraham @ Chelsio MFC after: 1 week Sponsored by: Chelsio Communications
* cxgbe/t4_tom: Remove duplicate unlock in t4_tom_deactivate.Navdeep Parhar2024-09-301-1/+0
| | | | | | Fixes: c1c524852f62 cxgbe/t4_tom: Implement uld_stop and uld_restart for ULD_TOM. MFC after: 1 week Sponsored by: Chelsio Communications
* cxgbe/t4_tom: Do not set unresolved entries to STALE in t4_l2t_update.Navdeep Parhar2024-09-241-1/+2
| | | | | | | | | An L2 entry in the driver's hash was marked STALE unconditionally if it changed in the kernel when its driver refcount is 0. Fix the driver to do this for VALID entries only. MFC after: 1 week Sponsored by: Chelsio Communications
* cxgbe/t4_tom: completely avoid L2T entries during stop/suspend.Navdeep Parhar2024-09-161-12/+13
| | | | | | | | | | | | 1. Mark the L2T entry valid only if t4_write_l2e succeeds, which won't happen if the adapter is stopped. This prevents L2T entries from sometimes getting (re)promoted to VALID on Tx activity during stop. 2. Discard a work request immediately instead of enqueueing it to the arp queue if the adapter is stopped. Fixes: c1c524852f62 cxgbe/t4_tom: Implement uld_stop and uld_restart for ULD_TOM. MFC after: 1 week Sponsored by: Chelsio Communications
* cxgbe/t4_tom: Demote STALE L2 table entries to RESOLVING in uld_stop.Navdeep Parhar2024-09-111-1/+1
| | | | | | | | | | | The STALE state means the L2T entry is valid in hardware but needs to be refreshed (ARP/NDP) in software. But stop/suspend wipes the hardware L2T and STALE entries need to be updated just like VALID entries to match actual hardware state. Fixes: c1c524852f62 cxgbe/t4_tom: Implement uld_stop and uld_restart for ULD_TOM. MFC after: 1 week Sponsored by: Chelsio Communications
* cxgbe/t4_tom: Add synq entry to the list before calling send_synack.Navdeep Parhar2024-09-061-4/+6
| | | | | | | | | | | This fixes a panic where the peer's ack to the synack arrives on a different queue and do_pass_establish tries to remove the synqe from synqe_list before it has been added by do_pass_accept_req. Reported by: Sony Arpita Das @ Chelsio Fixes: 283333c0e329 cxgbe/t4_tom: Track all synq entries in a per-adapter list. MFC after: 1 week Sponsored by: Chelsio Communications
* cxgbe(4): Stop checking for failures from malloc/mb_alloc_ext_pgs(M_WAITOK)Zhenlei Huang2024-09-031-5/+0
| | | | | MFC after: 1 week Differential Revision: https://reviews.freebsd.org/D45852
* cxgbe/t4_tom: bugfixes in stop/restart.Navdeep Parhar2024-08-053-23/+52
| | | | | | | | | 1. Remove toepcb from the toep_list on active open failure. 2. Purge the wr_list for an L2T entry on an adpater stop. Fixes: c1c524852f62 cxgbe/t4_tom: Implement uld_stop and uld_restart for ULD_TOM. MFC after: 2 weeks Sponsored by: Chelsio Communications
* Deprecate contigfree(9) in favour of free(9)Bjoern A. Zeeb2024-07-261-3/+3
| | | | | | | | | | | | | | | As of 9e6544dd6e02c46b805d11ab925c4f3b18ad7a4b contigfree(9) is no longer needed and should not be used anymore. We leave a wrapper for 3rd party code in at least 15.x but remove (almost) all other cases from the tree. This leaves one use of contigfree(9) untouched; that was the original trigger for 9e6544dd6e02 and is handled in D45813 (to be committed seperately later). Sponsored by: The FreeBSD Foundation Reviewed by: markj, kib Tested by: pho (10h stress test run) Differential Revision: https://reviews.freebsd.org/D46099
* cxgbe/t4_tom: Implement uld_stop and uld_restart for ULD_TOM.Navdeep Parhar2024-07-224-72/+408
| | | | | | | | | | | | | | | | | | | | This allows the adapter to be suspended or reset even when stateful TOE is active, in some limited configurations. The LLD has already stopped the adapter hardware and all its queues by the time these ULD routines get called. The general approach in t4_tom is to purge the lookup tables immediately so that they are ready for operation by the time the adapter resumes, and park all the resources left hanging by the stopped hardware into separate "stranded" queues that can be dealt with at leisure. Outstanding active opens, live connections, and synq entries (for connections in the middle of the 3-way handshake) are all treated as if the hardware had reported an abrupt error for the tid. The servers/listeners are a bit different in that no error is reported. They're just noted as non-functional when the hardware stops and are recreated by the driver during restart. MFC after: 1 month Sponsored by: Chelsio Communications
* cxgbe/t4_tom: Track all synq entries in a per-adapter list.Navdeep Parhar2024-07-223-0/+16
| | | | | | | | | | Live tid entries in tid_tab are either full fledged connections or synq entries. toep_list tracks the connections already and this change adds a synqe_list to track the synq entries. These two lists can be used to enumerate and iterate over all live tids. MFC after: 1 week Sponsored by: Chelsio Communications
* cxgbe(4): Stop and restart the L2T allocator with the LLD.Navdeep Parhar2024-07-221-0/+7
| | | | | | | | | | | | L2T entries are used by both filters and TOE and the L2T is shared between the base driver (LLD) and the TOM ULD. Add a flag to indicate that the L2T is stopped, which means: * t4_alloc_l2e and t4_l2t_alloc_switching will not allocate new entries. * t4_tom will ignore all ARP/NDP updates from the kernel. * Previously allocated L2T entries can still be freed. MFC after: 1 month Sponsored by: Chelsio Communications
* cxgbe(4): Changes to ULD list management.Navdeep Parhar2024-07-191-5/+4
| | | | | | | | | | | | | | | * Convert t4_uld_list to an array. There will be at most 3 items in the list and it's simpler to track them in an array with a fixed slot for each ULD. * There is no need to refcount ULDs so stop doing that. * Add uld_ prefix to all members of uld_info. * Rename async_event to uld_stop to match its actual purpose. Call it for all ULDs and not just ULD_IWARP. Reviewed by: jhb MFC after: 1 week Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D46029
* cxgbe(4): Make no assumptions about the start and size of the L2T table.Navdeep Parhar2024-07-191-8/+13
| | | | | | | | | | | Do not assume that the table starts at index 0 and is typically 4K in size. The only thing the driver needs to verify is that its use of F_SYNC_WR doesn't collide with the L2T hwidx range. Reviewed by: jhb MFC after: 1 week Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D46028
* cxgbe/t4_tom: Free up hardware resources when the final CPL is received.Navdeep Parhar2024-07-191-19/+39
| | | | | | | | | | | Final CPL means the tid is done in the hardware and other resources associated with it can be freed right away. There is no need to wait for the kernel to detach the toepcb. Reviewed by: jhb MFC after: 1 week Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D45991
* cxgbe/t4_tom: Detach the toep from the tcpcb when entering TIME_WAIT.Navdeep Parhar2024-07-193-1/+4
| | | | | | | | | | | | | | | The kernel used to call tod_pcb_detach when entering TIME_WAIT but that seems to have changed, likely with the TIME_WAIT overhaul in the kernel some time ago. Catch up by having the driver perform the detach. The hardware does not handle TIME_WAIT so it's important to detach and let the kernel arm the 2MSL timer to deal with it. Reported by: Sony Arpita Das @ Chelsio Reviewed by: jhb MFC after: 1 week Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D45990
* cxgbe(4): Consolidate all mk_set_tcb_field_ulp in one place.Navdeep Parhar2024-05-173-123/+21
| | | | | MFC after: 1 week Sponsored by: Chelsio Communications
* cxgbe/tom: Fix the rx channel selection in options2.Navdeep Parhar2024-05-011-2/+4
| | | | | | | | This affects TOE operation when multiple rx c-channels are in use for offload, which is an unusual configuration. MFC after: 1 week Sponsored by: Chelsio Communications
generated by cgit v1.2.3 (git 2.25.1) at 2026-04-05 20:05:03 +0000