aboutsummaryrefslogtreecommitdiff
path: root/sys/dev/ioat
Commit message (Collapse)AuthorAgeFilesLines
* ioat: Add Ice Lake ID.Alexander Motin2022-10-201-0/+2
| | | | | | MFC after: 2 weeks (cherry picked from commit e4e91353c62e2af425061a4e56e1f24dcca2dd0f)
* MFC r356216: Don't spin on cleanup_lock if we are not interrupt.Alexander Motin2020-01-071-1/+6
| | | | | | | | | | | If somebody else holds that lock, it will likely do the work for us. If it won't, then we return here later and retry. Under heavy load it allows to avoid lock congestion between interrupt and polling threads. Notes: svn path=/stable/12/; revision=356429
* MFC r354752: Cleanup address range checks in ioat(4).Alexander Motin2019-11-292-69/+26
| | | | | | | | | | | | | | | | | - Deduce allowed address range for bus_dma(9) from the hardware version. Different versions (CPU generations) have different documented limits. - Remove difference between address ranges for src/dst and crc. At least docs for few recent generations of CPUs do not mention anything like that, while older are already limited with above limits. - Remove address assertions from arguments. While I do not think the addresses out of allowed ranges should realistically happen there due to the platforms physical address limitations, there is now bus_dma(9) to make sure of that, preferably via IOMMU. - Since crc now has the same address range as src/dst, remove crc_dmamap, reusing dst2_dmamap instead. Notes: svn path=/stable/12/; revision=355200
* MFC r345813 (by tychon):Alexander Motin2019-11-292-15/+232
| | | | | | | | ioat(4) should use bus_dma(9) for the operation source and destination addresses Notes: svn path=/stable/12/; revision=355199
* MFC r345812 (by tychon):Alexander Motin2019-11-292-1/+28
| | | | | | | | ioatcontrol(8) could exercise 8k-aligned copy with page-break, crc and crc-copy modes. Notes: svn path=/stable/12/; revision=355197
* MFC r354753: Initialize *comp_update with valid value.Alexander Motin2019-11-291-6/+5
| | | | | | | | | I've noticed that sometimes with enabled DMAR initial write from device to this address is somehow getting delayed, triggering assertion due to zero default being invalid. Notes: svn path=/stable/12/; revision=355195
* MFC r354703: Pass more reasonable WAIT flags to bus_dma(9) calls.Alexander Motin2019-11-281-3/+4
| | | | Notes: svn path=/stable/12/; revision=355153
* MFC r354841: Add ioat_get_domain() to ioat(4) KPI.Alexander Motin2019-11-262-0/+10
| | | | | | | This allows NUMA-aware consumers to reduce inter-domain traffic. Notes: svn path=/stable/12/; revision=355111
* MFC r352787: Replace argument checks with assertions.Alexander Motin2019-10-151-83/+36
| | | | | | | | | Those functions are used by kernel, and we can't check all possible argument errors in production kernel. Plus according to docs many of those errors are checked by hardware. Assertions should just help with code debugging. Notes: svn path=/stable/12/; revision=353580
* MFC r352544: Improve ioat(4) NUMA-awareness.Alexander Motin2019-09-282-5/+14
| | | | | | | | | | | | Allocate ioat->ring memory from the device domain. Schedule ioat->poll_timer to the first CPU of the device domain. According to pcm-numa tool from intel-pcm port, this reduces number of remote DRAM accesses while copying data by 75%. And unless it is a noise, I've noticed some speed improvement when copying data to other domain. Notes: svn path=/stable/12/; revision=352827
* MFC r344441: Fix few issues in ioat(4) driver.Alexander Motin2019-02-282-228/+135
| | | | | | | | | | | | | | | | | - Do not explicitly count active descriptors. It allows hardware reset to happen while device is still referenced, plus simplifies locking. - Do not stop/start callout each time the queue becomes empty. Let it run to completion and rearm if needed, that is much cheaper then to touch it every time, plus also simplifies locking. - Decouple submit and cleanup locks, making driver reentrant. - Avoid memory mapped status register read on every interrupt. - Improve locking during device attach/detach. - Remove some no longer used variables. Sponsored by: iXsystems, Inc. Notes: svn path=/stable/12/; revision=344649
* MFC r343125 (by cem): ioat(4): Set __result_use_check on ioat_acquire_reserveAlexander Motin2019-02-211-1/+2
| | | | | | | | | | | Even M_WAITOK callers must check for failure. For example, if the device is quiescing, either due to automatic error-recovery induced reset, or due to administrative detach, the routine will return ENXIO and the acquire reference will not be held. So, there is no mode in which it is safe to assume the routine succeeds without checking. Notes: svn path=/stable/12/; revision=344400
* Reapply, with minor tweaks, r338025, from the original commit:Warner Losh2018-09-261-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Remove unused and easy to misuse PNP macro parameter Inspired by r338025, just remove the element size parameter to the MODULE_PNP_INFO macro entirely. The 'table' parameter is now required to have correct pointer (or array) type. Since all invocations of the macro already had this property and the emitted PNP data continues to include the element size, there is no functional change. Mostly done with the coccinelle 'spatch' tool: $ cat modpnpsize0.cocci @normaltables@ identifier b,c; expression a,d,e; declarer MODULE_PNP_INFO; @@ MODULE_PNP_INFO(a,b,c,d, -sizeof(d[0]), e); @singletons@ identifier b,c,d; expression a; declarer MODULE_PNP_INFO; @@ MODULE_PNP_INFO(a,b,c,&d, -sizeof(d), 1); $ rg -l MODULE_PNP_INFO -- sys | \ xargs spatch --in-place --sp-file modpnpsize0.cocci (Note that coccinelle invokes diff(1) via a PATH search and expects diff to tolerate the -B flag, which BSD diff does not. So I had to link gdiff into PATH as diff to use spatch.) Tinderbox'd (-DMAKE_JUST_KERNELS). Approved by: re (glen) Notes: svn path=/head/; revision=338948
* Back out r338035 until Warner is finished churning GSoC PNP patchesConrad Meyer2018-08-191-1/+1
| | | | | | | | | | I was not aware Warner was making or planning to make forward progress in this area and have since been informed of that. It's easy to apply/reapply when churn dies down. Notes: svn path=/head/; revision=338037
* Remove unused and easy to misuse PNP macro parameterConrad Meyer2018-08-191-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Inspired by r338025, just remove the element size parameter to the MODULE_PNP_INFO macro entirely. The 'table' parameter is now required to have correct pointer (or array) type. Since all invocations of the macro already had this property and the emitted PNP data continues to include the element size, there is no functional change. Mostly done with the coccinelle 'spatch' tool: $ cat modpnpsize0.cocci @normaltables@ identifier b,c; expression a,d,e; declarer MODULE_PNP_INFO; @@ MODULE_PNP_INFO(a,b,c,d, -sizeof(d[0]), e); @singletons@ identifier b,c,d; expression a; declarer MODULE_PNP_INFO; @@ MODULE_PNP_INFO(a,b,c,&d, -sizeof(d), 1); $ rg -l MODULE_PNP_INFO -- sys | \ xargs spatch --in-place --sp-file modpnpsize0.cocci (Note that coccinelle invokes diff(1) via a PATH search and expects diff to tolerate the -B flag, which BSD diff does not. So I had to link gdiff into PATH as diff to use spatch.) Tinderbox'd (-DMAKE_JUST_KERNELS). Notes: svn path=/head/; revision=338035
* Use '#' rather than some made up name for fields we want to ignore.Warner Losh2017-12-221-1/+1
| | | | Notes: svn path=/head/; revision=327089
* ioat(4): Add Skylake Xeon PCI-IDConrad Meyer2017-12-051-0/+2
| | | | | | | | | | SKX IOAT is just another 3.2 version of the CBDMA engine. Submitted by: Deepak Veliath <deepak.veliath AT isilon.com> Sponsored by: Dell EMC Isilon Notes: svn path=/head/; revision=326572
* Add PNP metadata to a few driversConrad Meyer2017-09-141-3/+4
| | | | | | | | | | | | An eventual devd(8) or other component should be able to scan buses and automatically load drivers that match device ids described in this metadata. Reviewed by: imp Sponsored by: Dell EMC Isilon Differential Revision: https://reviews.freebsd.org/D12364 Notes: svn path=/head/; revision=323586
* ioat: don't specify inline for function with variable argument listAndriy Gapon2017-03-041-2/+2
| | | | | | | | | | | Modern GCC and Clang simply ignore the qualifier, while the old base GCC produces a warning (treated as an error in the kernel build). Approved by: cem MFC after: 5 days Notes: svn path=/head/; revision=314666
* ioat(4): Compile on i386Conrad Meyer2016-12-041-2/+3
| | | | | | | | | | | | Truncate BUS_SPACE_MAXADDR_40BIT to essentially BUS_SPACE_MAXADDR_32BIT on platforms with 32-bit bus_addr_t (i.e., i386). PR: 215034 Reported by: ngie@ Sponsored by: Dell EMC Isilon Notes: svn path=/head/; revision=309526
* ioat(4): Fix 'bogus completion_pending' KASSERTConrad Meyer2016-11-302-5/+10
| | | | | | | | | | | | Fix ioat_release to only set is_completion_pending if DMAs were actually queued. Otherwise, the spurious flag could trigger an assert in the reset path on INVARIANTS kernels. Reviewed by: bdrewery, Suraj Raju @ Isilon Sponsored by: Dell EMC Isilon Notes: svn path=/head/; revision=309338
* ioat(4): Fix race between process_events and reset_hwConrad Meyer2016-11-111-0/+9
| | | | | | | | | | | | | | | | | In the case where a hardware error is detected during ioat_process_events, hardware may advance (by one descriptor, probably) and a subsequent ioat_process_events may race the intended ioat_reset_hw followup. In that case, the second process_events would observe a completion update that does not match the software "last_seen" status, and attempt to successfully complete already-failed descriptors. Guard against this race with the resetting_cleanup flag. Reviewed by: bdrewery, markj Sponsored by: Dell EMC Isilon Notes: svn path=/head/; revision=308553
* ioat(4): Read CHANSTS register for suspended/halted checksConrad Meyer2016-11-021-0/+6
| | | | | | | | | | | | | The device doesn't accurately update the CHANCMP address with the device state when the device is suspended or halted. So, read the CHANSTS register to check for those states. We still need to read the CHANCMP address for the last completed descriptor. Sponsored by: Dell EMC Isilon Notes: svn path=/head/; revision=308230
* ioat(4): Allocate contiguous descriptorsConrad Meyer2016-11-012-125/+93
| | | | | | | | | | | | This allows us to make strong assertions about descriptor address validity. Additionally, future generations of the ioat(4) hardware will require contiguous descriptors. Reviewed by: markj Sponsored by: Dell EMC Isilon Notes: svn path=/head/; revision=308179
* ioat(4): Simplify by removing dynamic scalingConrad Meyer2016-11-012-327/+13
| | | | | | | | | | | | | | | | | This paves the way for a contiguous descriptor array. A contiguous descriptor array has the benefit that we can make strong assertions about whether an address is a valid descriptor or not. The other benefit is that future generations of I/OAT hardware will require a contiguous descriptor array anyway. The downside is that after system boot, big chunks of contiguous memory is much harder to find. So dynamic scaling after boot is basically impossible. Reviewed by: markj Sponsored by: Dell EMC Isilon Notes: svn path=/head/; revision=308178
* ioat(4): Use memory completion rather than device registerConrad Meyer2016-10-282-2/+11
| | | | | | | | | | | | | | | | The CHANSTS register is a split 64-bit register on CBDMA units before hardware v3.3. If a torn read happens during ioat_process_events(), software cannot know when to stop completing descriptors correctly. So, just use the device-pushed main memory channel status instead. Remove the ioat_get_active() seatbelt as well. It does nothing if the completion address is valid. Sponsored by: Dell EMC Isilon Notes: svn path=/head/; revision=308070
* ioat(4): Add failpoint for delay() in ioat_releaseConrad Meyer2016-10-281-2/+7
| | | | | | | Sponsored by: Dell EMC Isilon Notes: svn path=/head/; revision=308069
* ioat(4): Assert the submit lock in ioat_submit_singleConrad Meyer2016-10-281-0/+2
| | | | | | | Sponsored by: Dell EMC Isilon Notes: svn path=/head/; revision=308068
* ioat(4): Add additional tracingConrad Meyer2016-10-281-6/+22
| | | | | | | | | These probes help track down driver bugs. Sponsored by: Dell EMC Isilon Notes: svn path=/head/; revision=308067
* ioat(4): Start poll timer when descriptors are released to HWConrad Meyer2016-09-111-7/+7
| | | | | | | | | Rather than when the software creates the descriptors. Sponsored by: Dell EMC Isilon Notes: svn path=/head/; revision=305711
* ioat(4): De-spam ioat_process_events KTR logsConrad Meyer2016-09-111-4/+2
| | | | | | | Sponsored by: Dell EMC Isilon Notes: svn path=/head/; revision=305710
* ioat(4): Despam relatively common hardware reset messagesConrad Meyer2016-09-011-2/+2
| | | | | | | Reported by: ngie@ Notes: svn path=/head/; revision=305259
* ioat(4): Add additional CTR tracing during resetConrad Meyer2016-08-291-1/+15
| | | | Notes: svn path=/head/; revision=305028
* ioat(4): Don't "complete" DMA descriptors prematurelyConrad Meyer2016-08-291-9/+19
| | | | | | | | | | | | | | | | In r304602, I mistakenly removed the ioat_process_events check that we weren't processing events before the hardware had completed the descriptor ("last_seen"). Reinstate that logic. Keep the defensive loop condition and additionally make sure we've actually completed a descriptor before blindly chasing the ring around. In reset, queue and finish the startup command before allowing any event processing or submission to occur. Avoid potential missed callouts by requeueing the poll later. Notes: svn path=/head/; revision=305027
* ioat(4): Allow callouts to be scheduled after hw resetConrad Meyer2016-08-221-0/+8
| | | | | | | | | | | | | | is_completion_pending governs whether or not a callout will be scheduled when new work is queued on the IOAT device. If true, a callout is already scheduled, so we do not need a new one. If false, we schedule one and set it true. Because resetting the hardware completed all outstanding work but failed to clear is_completion_pending, no new callout could be scheduled after a reset with pending work. This resulted in a driver hang for polled-only work. Notes: svn path=/head/; revision=304603
* ioat(4): Don't process events past queue headConrad Meyer2016-08-221-19/+8
| | | | | | | | Fix a race where the completion routine could overrun the active ring area in some situations. Notes: svn path=/head/; revision=304602
* ioat(4): Log channel number in CTR eventsConrad Meyer2016-08-051-16/+18
| | | | Notes: svn path=/head/; revision=303761
* ioat(4): Check ring links at grow/shrink in INVARIANTSConrad Meyer2016-07-121-0/+24
| | | | Notes: svn path=/head/; revision=302686
* ioat(4): Add KTR trace for ioat_reset_hwConrad Meyer2016-07-121-0/+2
| | | | Notes: svn path=/head/; revision=302685
* ioat(4): Enhance KTR logging for descriptor completionsConrad Meyer2016-07-121-2/+4
| | | | Notes: svn path=/head/; revision=302684
* ioat(4): Assert against ring underflowConrad Meyer2016-07-121-0/+5
| | | | Notes: svn path=/head/; revision=302683
* ioat_reserve_space: Recheck quiescing flag after dropping submit lockConrad Meyer2016-07-121-5/+8
| | | | | | | | Fix a minor bound check error while here (ring can only hold 1 << MAX_ORDER - 1 entries). Notes: svn path=/head/; revision=302682
* ioat(4): Remove force_hw_error sysctl; it does not work reliablyConrad Meyer2016-07-121-35/+0
| | | | Notes: svn path=/head/; revision=302681
* ioat(4): Export HW capabilities to consumersConrad Meyer2016-07-123-21/+40
| | | | Notes: svn path=/head/; revision=302680
* ioat(4): Submitters pick up a shovel if queue is too fullConrad Meyer2016-07-122-0/+25
| | | | | | | Before attempting to grow the ring. Notes: svn path=/head/; revision=302679
* ioat(4): Don't shrink ring if activeConrad Meyer2016-07-121-2/+5
| | | | Notes: svn path=/head/; revision=302678
* ioat(4): Print some more useful information about the ring from ddb "show ioat"Conrad Meyer2016-07-121-1/+30
| | | | Notes: svn path=/head/; revision=302677
* ioat(4): Shrink using the correct timerConrad Meyer2016-07-121-1/+1
| | | | | | | | | Fix a typo introduced in r302352. Sponsored by: EMC / Isilon Storage Division Notes: svn path=/head/; revision=302669
* ioat(4): Block asynchronous work during HW resetConrad Meyer2016-07-052-10/+69
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Fix the race between ioat_reset_hw and ioat_process_events. HW reset isn't protected by a lock because it can sleep for a long time (40.1 ms). This resulted in a race where we would process bogus parts of the descriptor ring as if it had completed. This looked like duplicate completions on old events, if your ring had looped at least once. Block callout and interrupt work while reset runs so the completion end of things does not observe indeterminate state and process invalid parts of the ring. Start the channel with a manually implemented ioat_null() to keep other submitters quiesced while we wait for the channel to start (100 us). r295605 may have made the race between ioat_reset_hw and ioat_process_events wider, but I believe it already existed before that revision. ioat_process_events can be invoked by two asynchronous sources: callout (softclock) and device interrupt. Those could race each other, to the same effect. Reviewed by: markj Approved by: re Sponsored by: EMC / Isilon Storage Division Differential Revision: https://reviews.freebsd.org/D7097 Notes: svn path=/head/; revision=302354
* ioat(4): Serialize ioat_reset_hw invocationsConrad Meyer2016-07-052-0/+14
| | | | | | | | | | Reviewed by: markj Approved by: re Sponsored by: EMC / Isilon Storage Division Differential Revision: https://reviews.freebsd.org/D7097 Notes: svn path=/head/; revision=302353
generated by cgit v1.2.3 (git 2.25.1) at 2026-04-14 16:53:38 +0000