aboutsummaryrefslogtreecommitdiff
path: root/sys/dev/ioat
Commit message (Collapse)AuthorAgeFilesLines
* ioat: plug set-but-not-unused varsMateusz Guzik2021-12-091-5/+2
| | | | Sponsored by: Rubicon Communications, LLC ("Netgate")
* ioat(4): Remove Giant from ioat_test enable/disable.Alexander Motin2021-09-042-16/+35
| | | | MFC after: 1 month
* Mark some sysctls as CTLFLAG_MPSAFE.Alexander Motin2021-08-111-3/+3
| | | | MFC after: 2 weeks
* Remove free_domain() and uma_zfree_domain().Mark Johnston2020-08-041-1/+1
| | | | | | | | | | | | | | These functions were introduced before UMA started ensuring that freed memory gets placed in domain-local caches. They no longer serve any purpose since UMA now provides their functionality by default. Remove them to simplyify the kernel memory allocator interfaces a bit. Reviewed by: cem, kib Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D25937 Notes: svn path=/head/; revision=363834
* Mark more nodes as CTLFLAG_MPSAFE or CTLFLAG_NEEDGIANT (17 of many)Pawel Biernacki2020-02-262-14/+17
| | | | | | | | | | | | | | | | | | | r357614 added CTLFLAG_NEEDGIANT to make it easier to find nodes that are still not MPSAFE (or already are but aren’t properly marked). Use it in preparation for a general review of all nodes. This is non-functional change that adds annotations to SYSCTL_NODE and SYSCTL_PROC nodes using one of the soon-to-be-required flags. Mark all obvious cases as MPSAFE. All entries that haven't been marked as MPSAFE before are by default marked as NEEDGIANT Approved by: kib (mentor, blanket) Commented by: kib, gallatin, melifaro Differential Revision: https://reviews.freebsd.org/D23718 Notes: svn path=/head/; revision=358333
* Don't spin on cleanup_lock if we are not interrupt.Alexander Motin2019-12-311-1/+6
| | | | | | | | | | | | | | If somebody else holds that lock, it will likely do the work for us. If it won't, then we return here later and retry. Under heavy load it allows to avoid lock congestion between interrupt and polling threads. MFC after: 1 week Sponsored by: iXsystems, Inc. Notes: svn path=/head/; revision=356216
* Add ioat_get_domain() to ioat(4) KPI.Alexander Motin2019-11-192-0/+10
| | | | | | | | | | This allows NUMA-aware consumers to reduce inter-domain traffic. MFC after: 1 week Sponsored by: iXsystems, Inc. Notes: svn path=/head/; revision=354841
* Initialize *comp_update with valid value.Alexander Motin2019-11-151-6/+5
| | | | | | | | | | | | I've noticed that sometimes with enabled DMAR initial write from device to this address is somehow getting delayed, triggering assertion due to zero default being invalid. MFC after: 2 weeks Sponsored by: iXsystems, Inc. Notes: svn path=/head/; revision=354753
* Cleanup address range checks in ioat(4).Alexander Motin2019-11-152-69/+26
| | | | | | | | | | | | | | | | | | | | | - Deduce allowed address range for bus_dma(9) from the hardware version. Different versions (CPU generations) have different documented limits. - Remove difference between address ranges for src/dst and crc. At least docs for few recent generations of CPUs do not mention anything like that, while older are already limited with above limits. - Remove address assertions from arguments. While I do not think the addresses out of allowed ranges should realistically happen there due to the platforms physical address limitations, there is now bus_dma(9) to make sure of that, preferably via IOMMU. - Since crc now has the same address range as src/dst, remove crc_dmamap, reusing dst2_dmamap instead. Discussed with: cem MFC after: 2 weeks Sponsored by: iXsystems, Inc. Notes: svn path=/head/; revision=354752
* Pass more reasonable WAIT flags to bus_dma(9) calls.Alexander Motin2019-11-141-3/+4
| | | | | | | MFC after: 2 weeks Notes: svn path=/head/; revision=354703
* Add CTLFLAG_STATS to the dev.ioat.N.stats sysctl OIDsEric van Gyzen2019-10-091-12/+12
| | | | | | | | | | Refer to r353111. MFC after: 2 weeks Sponsored by: Dell EMC Isilon Notes: svn path=/head/; revision=353344
* Replace argument checks with assertions.Alexander Motin2019-09-271-83/+36
| | | | | | | | | | | Those functions are used by kernel, and we can't check all possible argument errors in production kernel. Plus according to docs many of those errors are checked by hardware. Assertions should just help with code debugging. MFC after: 2 weeks Notes: svn path=/head/; revision=352787
* Improve ioat(4) NUMA-awareness.Alexander Motin2019-09-192-5/+14
| | | | | | | | | | | | | | | Allocate ioat->ring memory from the device domain. Schedule ioat->poll_timer to the first CPU of the device domain. According to pcm-numa tool from intel-pcm port, this reduces number of remote DRAM accesses while copying data by 75%. And unless it is a noise, I've noticed some speed improvement when copying data to other domain. MFC after: 1 week Sponsored by: iXsystems, Inc. Notes: svn path=/head/; revision=352544
* Extract eventfilter declarations to sys/_eventfilter.hConrad Meyer2019-05-201-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | This allows replacing "sys/eventfilter.h" includes with "sys/_eventfilter.h" in other header files (e.g., sys/{bus,conf,cpu}.h) and reduces header pollution substantially. EVENTHANDLER_DECLARE and EVENTHANDLER_LIST_DECLAREs were moved out of .c files into appropriate headers (e.g., sys/proc.h, powernv/opal.h). As a side effect of reduced header pollution, many .c files and headers no longer contain needed definitions. The remainder of the patch addresses adding appropriate includes to fix those files. LOCK_DEBUG and LOCK_FILE_LINE_ARG are moved to sys/_lock.h, as required by sys/mutex.h since r326106 (but silently protected by header pollution prior to this change). No functional change (intended). Of course, any out of tree modules that relied on header pollution for sys/eventhandler.h, sys/lock.h, or sys/mutex.h inclusion need to be fixed. __FreeBSD_version has been bumped. Notes: svn path=/head/; revision=347984
* ioat(4) should use bus_dma(9) for the operation source and destinationTycho Nightingale2019-04-022-15/+232
| | | | | | | | | | | addresses Reviewed by: cem Sponsored by: Dell EMC Isilon Differential Revision: https://reviews.freebsd.org/D19725 Notes: svn path=/head/; revision=345813
* ioatcontrol(8) could exercise 8k-aligned copy with page-break, crc andTycho Nightingale2019-04-022-1/+28
| | | | | | | | | | | crc-copy modes. Reviewed by: cem Sponsored by: Dell EMC Isilon Differential Revision: https://reviews.freebsd.org/D19780 Notes: svn path=/head/; revision=345812
* Fix few issues in ioat(4) driver.Alexander Motin2019-02-212-228/+135
| | | | | | | | | | | | | | | | | | | | - Do not explicitly count active descriptors. It allows hardware reset to happen while device is still referenced, plus simplifies locking. - Do not stop/start callout each time the queue becomes empty. Let it run to completion and rearm if needed, that is much cheaper then to touch it every time, plus also simplifies locking. - Decouple submit and cleanup locks, making driver reentrant. - Avoid memory mapped status register read on every interrupt. - Improve locking during device attach/detach. - Remove some no longer used variables. Reviewed by: cem MFC after: 1 week Sponsored by: iXsystems, Inc. Differential Revision: https://reviews.freebsd.org/D19231 Notes: svn path=/head/; revision=344441
* ioat(4): Set __result_use_check on ioat_acquire_reserveConrad Meyer2019-01-171-1/+2
| | | | | | | | | | | | | Even M_WAITOK callers must check for failure. For example, if the device is quiescing, either due to automatic error-recovery induced reset, or due to administrative detach, the routine will return ENXIO and the acquire reference will not be held. So, there is no mode in which it is safe to assume the routine succeeds without checking. Sponsored by: Dell EMC Isilon Notes: svn path=/head/; revision=343125
* Reapply, with minor tweaks, r338025, from the original commit:Warner Losh2018-09-261-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Remove unused and easy to misuse PNP macro parameter Inspired by r338025, just remove the element size parameter to the MODULE_PNP_INFO macro entirely. The 'table' parameter is now required to have correct pointer (or array) type. Since all invocations of the macro already had this property and the emitted PNP data continues to include the element size, there is no functional change. Mostly done with the coccinelle 'spatch' tool: $ cat modpnpsize0.cocci @normaltables@ identifier b,c; expression a,d,e; declarer MODULE_PNP_INFO; @@ MODULE_PNP_INFO(a,b,c,d, -sizeof(d[0]), e); @singletons@ identifier b,c,d; expression a; declarer MODULE_PNP_INFO; @@ MODULE_PNP_INFO(a,b,c,&d, -sizeof(d), 1); $ rg -l MODULE_PNP_INFO -- sys | \ xargs spatch --in-place --sp-file modpnpsize0.cocci (Note that coccinelle invokes diff(1) via a PATH search and expects diff to tolerate the -B flag, which BSD diff does not. So I had to link gdiff into PATH as diff to use spatch.) Tinderbox'd (-DMAKE_JUST_KERNELS). Approved by: re (glen) Notes: svn path=/head/; revision=338948
* Back out r338035 until Warner is finished churning GSoC PNP patchesConrad Meyer2018-08-191-1/+1
| | | | | | | | | | I was not aware Warner was making or planning to make forward progress in this area and have since been informed of that. It's easy to apply/reapply when churn dies down. Notes: svn path=/head/; revision=338037
* Remove unused and easy to misuse PNP macro parameterConrad Meyer2018-08-191-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Inspired by r338025, just remove the element size parameter to the MODULE_PNP_INFO macro entirely. The 'table' parameter is now required to have correct pointer (or array) type. Since all invocations of the macro already had this property and the emitted PNP data continues to include the element size, there is no functional change. Mostly done with the coccinelle 'spatch' tool: $ cat modpnpsize0.cocci @normaltables@ identifier b,c; expression a,d,e; declarer MODULE_PNP_INFO; @@ MODULE_PNP_INFO(a,b,c,d, -sizeof(d[0]), e); @singletons@ identifier b,c,d; expression a; declarer MODULE_PNP_INFO; @@ MODULE_PNP_INFO(a,b,c,&d, -sizeof(d), 1); $ rg -l MODULE_PNP_INFO -- sys | \ xargs spatch --in-place --sp-file modpnpsize0.cocci (Note that coccinelle invokes diff(1) via a PATH search and expects diff to tolerate the -B flag, which BSD diff does not. So I had to link gdiff into PATH as diff to use spatch.) Tinderbox'd (-DMAKE_JUST_KERNELS). Notes: svn path=/head/; revision=338035
* Use '#' rather than some made up name for fields we want to ignore.Warner Losh2017-12-221-1/+1
| | | | Notes: svn path=/head/; revision=327089
* ioat(4): Add Skylake Xeon PCI-IDConrad Meyer2017-12-051-0/+2
| | | | | | | | | | SKX IOAT is just another 3.2 version of the CBDMA engine. Submitted by: Deepak Veliath <deepak.veliath AT isilon.com> Sponsored by: Dell EMC Isilon Notes: svn path=/head/; revision=326572
* Add PNP metadata to a few driversConrad Meyer2017-09-141-3/+4
| | | | | | | | | | | | An eventual devd(8) or other component should be able to scan buses and automatically load drivers that match device ids described in this metadata. Reviewed by: imp Sponsored by: Dell EMC Isilon Differential Revision: https://reviews.freebsd.org/D12364 Notes: svn path=/head/; revision=323586
* ioat: don't specify inline for function with variable argument listAndriy Gapon2017-03-041-2/+2
| | | | | | | | | | | Modern GCC and Clang simply ignore the qualifier, while the old base GCC produces a warning (treated as an error in the kernel build). Approved by: cem MFC after: 5 days Notes: svn path=/head/; revision=314666
* ioat(4): Compile on i386Conrad Meyer2016-12-041-2/+3
| | | | | | | | | | | | Truncate BUS_SPACE_MAXADDR_40BIT to essentially BUS_SPACE_MAXADDR_32BIT on platforms with 32-bit bus_addr_t (i.e., i386). PR: 215034 Reported by: ngie@ Sponsored by: Dell EMC Isilon Notes: svn path=/head/; revision=309526
* ioat(4): Fix 'bogus completion_pending' KASSERTConrad Meyer2016-11-302-5/+10
| | | | | | | | | | | | Fix ioat_release to only set is_completion_pending if DMAs were actually queued. Otherwise, the spurious flag could trigger an assert in the reset path on INVARIANTS kernels. Reviewed by: bdrewery, Suraj Raju @ Isilon Sponsored by: Dell EMC Isilon Notes: svn path=/head/; revision=309338
* ioat(4): Fix race between process_events and reset_hwConrad Meyer2016-11-111-0/+9
| | | | | | | | | | | | | | | | | In the case where a hardware error is detected during ioat_process_events, hardware may advance (by one descriptor, probably) and a subsequent ioat_process_events may race the intended ioat_reset_hw followup. In that case, the second process_events would observe a completion update that does not match the software "last_seen" status, and attempt to successfully complete already-failed descriptors. Guard against this race with the resetting_cleanup flag. Reviewed by: bdrewery, markj Sponsored by: Dell EMC Isilon Notes: svn path=/head/; revision=308553
* ioat(4): Read CHANSTS register for suspended/halted checksConrad Meyer2016-11-021-0/+6
| | | | | | | | | | | | | The device doesn't accurately update the CHANCMP address with the device state when the device is suspended or halted. So, read the CHANSTS register to check for those states. We still need to read the CHANCMP address for the last completed descriptor. Sponsored by: Dell EMC Isilon Notes: svn path=/head/; revision=308230
* ioat(4): Allocate contiguous descriptorsConrad Meyer2016-11-012-125/+93
| | | | | | | | | | | | This allows us to make strong assertions about descriptor address validity. Additionally, future generations of the ioat(4) hardware will require contiguous descriptors. Reviewed by: markj Sponsored by: Dell EMC Isilon Notes: svn path=/head/; revision=308179
* ioat(4): Simplify by removing dynamic scalingConrad Meyer2016-11-012-327/+13
| | | | | | | | | | | | | | | | | This paves the way for a contiguous descriptor array. A contiguous descriptor array has the benefit that we can make strong assertions about whether an address is a valid descriptor or not. The other benefit is that future generations of I/OAT hardware will require a contiguous descriptor array anyway. The downside is that after system boot, big chunks of contiguous memory is much harder to find. So dynamic scaling after boot is basically impossible. Reviewed by: markj Sponsored by: Dell EMC Isilon Notes: svn path=/head/; revision=308178
* ioat(4): Use memory completion rather than device registerConrad Meyer2016-10-282-2/+11
| | | | | | | | | | | | | | | | The CHANSTS register is a split 64-bit register on CBDMA units before hardware v3.3. If a torn read happens during ioat_process_events(), software cannot know when to stop completing descriptors correctly. So, just use the device-pushed main memory channel status instead. Remove the ioat_get_active() seatbelt as well. It does nothing if the completion address is valid. Sponsored by: Dell EMC Isilon Notes: svn path=/head/; revision=308070
* ioat(4): Add failpoint for delay() in ioat_releaseConrad Meyer2016-10-281-2/+7
| | | | | | | Sponsored by: Dell EMC Isilon Notes: svn path=/head/; revision=308069
* ioat(4): Assert the submit lock in ioat_submit_singleConrad Meyer2016-10-281-0/+2
| | | | | | | Sponsored by: Dell EMC Isilon Notes: svn path=/head/; revision=308068
* ioat(4): Add additional tracingConrad Meyer2016-10-281-6/+22
| | | | | | | | | These probes help track down driver bugs. Sponsored by: Dell EMC Isilon Notes: svn path=/head/; revision=308067
* ioat(4): Start poll timer when descriptors are released to HWConrad Meyer2016-09-111-7/+7
| | | | | | | | | Rather than when the software creates the descriptors. Sponsored by: Dell EMC Isilon Notes: svn path=/head/; revision=305711
* ioat(4): De-spam ioat_process_events KTR logsConrad Meyer2016-09-111-4/+2
| | | | | | | Sponsored by: Dell EMC Isilon Notes: svn path=/head/; revision=305710
* ioat(4): Despam relatively common hardware reset messagesConrad Meyer2016-09-011-2/+2
| | | | | | | Reported by: ngie@ Notes: svn path=/head/; revision=305259
* ioat(4): Add additional CTR tracing during resetConrad Meyer2016-08-291-1/+15
| | | | Notes: svn path=/head/; revision=305028
* ioat(4): Don't "complete" DMA descriptors prematurelyConrad Meyer2016-08-291-9/+19
| | | | | | | | | | | | | | | | In r304602, I mistakenly removed the ioat_process_events check that we weren't processing events before the hardware had completed the descriptor ("last_seen"). Reinstate that logic. Keep the defensive loop condition and additionally make sure we've actually completed a descriptor before blindly chasing the ring around. In reset, queue and finish the startup command before allowing any event processing or submission to occur. Avoid potential missed callouts by requeueing the poll later. Notes: svn path=/head/; revision=305027
* ioat(4): Allow callouts to be scheduled after hw resetConrad Meyer2016-08-221-0/+8
| | | | | | | | | | | | | | is_completion_pending governs whether or not a callout will be scheduled when new work is queued on the IOAT device. If true, a callout is already scheduled, so we do not need a new one. If false, we schedule one and set it true. Because resetting the hardware completed all outstanding work but failed to clear is_completion_pending, no new callout could be scheduled after a reset with pending work. This resulted in a driver hang for polled-only work. Notes: svn path=/head/; revision=304603
* ioat(4): Don't process events past queue headConrad Meyer2016-08-221-19/+8
| | | | | | | | Fix a race where the completion routine could overrun the active ring area in some situations. Notes: svn path=/head/; revision=304602
* ioat(4): Log channel number in CTR eventsConrad Meyer2016-08-051-16/+18
| | | | Notes: svn path=/head/; revision=303761
* ioat(4): Check ring links at grow/shrink in INVARIANTSConrad Meyer2016-07-121-0/+24
| | | | Notes: svn path=/head/; revision=302686
* ioat(4): Add KTR trace for ioat_reset_hwConrad Meyer2016-07-121-0/+2
| | | | Notes: svn path=/head/; revision=302685
* ioat(4): Enhance KTR logging for descriptor completionsConrad Meyer2016-07-121-2/+4
| | | | Notes: svn path=/head/; revision=302684
* ioat(4): Assert against ring underflowConrad Meyer2016-07-121-0/+5
| | | | Notes: svn path=/head/; revision=302683
* ioat_reserve_space: Recheck quiescing flag after dropping submit lockConrad Meyer2016-07-121-5/+8
| | | | | | | | Fix a minor bound check error while here (ring can only hold 1 << MAX_ORDER - 1 entries). Notes: svn path=/head/; revision=302682
* ioat(4): Remove force_hw_error sysctl; it does not work reliablyConrad Meyer2016-07-121-35/+0
| | | | Notes: svn path=/head/; revision=302681
* ioat(4): Export HW capabilities to consumersConrad Meyer2016-07-123-21/+40
| | | | Notes: svn path=/head/; revision=302680