aboutsummaryrefslogtreecommitdiff
path: root/sys/dev/iommu
Commit message (Collapse)AuthorAgeFilesLines
* iommu_gas: Fix assertion.Ruslan Bukin2026-02-191-1/+1
| | | | | | | | placeholder entry end changed to 0 in f591287756368 ("iommu_gas: make placeholder entry at the start of the GAS zero size") Reviewed by: kib Differential Revision: https://reviews.freebsd.org/D55326
* AMD IOMMU: fix per-device IOMMU bypass when IR is enabledJason A. Harmening2025-10-201-2/+5
| | | | | | | | | | | | | | | | | | | | | | | | When interrupt remapping (IR) is enabled, the device table entry (DTE) for a given device will likely be initialized by amdiommu_ir_find() during MSI configuration. This function directly calls amdiommu_get_ctx_for_dev() with id_mapped=false, which means that any attempt to disable DMA remapping for the device (e.g. by setting hw.busdma.pciD.B.S.F='bounce' in the loader tunables) will effectively be ignored as the paging mode field in the DTE will not be set to 0 as required for identity mapping. This will ultimately produce an unusable device, because busdma will later observe the bounce configuration through iommu_instantiate_ctx() and will employ the non-translated 'bounce' busdma methods for the device, while the DTE remains configured to enable translation. Fix this by tweaking iommu_instantiate_ctx() to always return the relevant per-device context object even if translation is disabled, and adopt it in amdiommu_ir_find() instead of directly calling amdiommu_get_ctx_for_dev(). Reviewed by: kib MFC after: 1 week Differential Revision: https://reviews.freebsd.org/D53209
* iommu_gas: Use proper prototype for SYSINIT functionsZhenlei Huang2025-10-131-1/+1
| | | | MFC after: 1 week
* iommu_get_requester(): make it more resilient against arbitrary dev argKonstantin Belousov2025-09-092-14/+42
| | | | | | | | | | | | | | | | | If passed the parent of a device instead of the device, the loop might end up with the host bridge in the pci local variable. If the passed device is not from the pci hierarchy, any of the calculated parents might be NULL. Change the interface to allow the caller to receive error. Instead of asserting, just issue a message and return ENXIO, allowing the caller to select appropriate action. PR: 289318 Reviewed by: jah Sponsored by: The FreeBSD Foundation MFC after: 2 weeks Differential revision: https://reviews.freebsd.org/D52406
* iommu_gas_map_region(): add comment explaining the ma array shiftKonstantin Belousov2025-04-011-0/+5
| | | | | | | | and the difference between start and entry->start values at this point. Sponsored by: Advanced Micro Devices (AMD) Sponsored by: The FreeBSD Foundation MFC after: 3 days
* iommu: eliminate iommu_free_ctx()Konstantin Belousov2024-11-031-2/+8
| | | | | | | | iommu_free_ctx_locked() alone is enough Sponsored by: Advanced Micro Devices (AMD) Sponsored by: The FreeBSD Foundation MFC after: 1 week
* iommu: move context link and ref count into device-independent partsKonstantin Belousov2024-10-131-0/+3
| | | | | | | | | This also allows to move some bits of ddb print routines into iommu_utils.c common for x86 iommu drivers. Sponsored by: Advanced Micro Devices (AMD) Sponsored by: The FreeBSD Foundation MFC after: 1 week
* iommu_get_requester(): do not panic if asked about non-pci deviceKonstantin Belousov2024-10-131-0/+6
| | | | | | | | | | For now, return zero rid and the device itself. Add a comment noting that eventually ACPI HID can be used to calculate rid for some more devices. Sponsored by: Advanced Micro Devices (AMD) Sponsored by: The FreeBSD Foundation MFC after: 1 week
* iommu.h: improve header self-sufficiencyKonstantin Belousov2024-10-131-0/+2
| | | | | | | | | The header embeds struct task into defined structures. Also it needs the PCI_BUSMAX constant. Sponsored by: Advanced Micro Devices (AMD) Sponsored by: The FreeBSD Foundation MFC after: 1 week
* iommu: extend iommu_map_entry to store the list of associated freed page ↵Konstantin Belousov2024-09-272-3/+14
| | | | | | | | | | | table pages The pages are inserted into the added slist if the entry parameter is passed to iommu_pgfree(). For now it is nop. Sponsored by: Advanced Micro Devices (AMD) Sponsored by: The FreeBSD Foundation MFC after: 1 week
* iommu: change iommu_domain_map_ops to take iommu_map_entryKonstantin Belousov2024-09-272-11/+9
| | | | | | | | instead of base/size. Sponsored by: Advanced Micro Devices (AMD) Sponsored by: The FreeBSD Foundation MFC after: 1 week
* iommu: add per-unit sysctls reporting the state of DMA and interrupt remappingKonstantin Belousov2024-09-272-0/+5
| | | | | | Sponsored by: Advanced Micro Devices (AMD) Sponsored by: The FreeBSD Foundation MFC after: 1 week
* busdma_iommu: indirect dmar-specific method calls in iommu_get_dev_ctx()Konstantin Belousov2024-09-042-5/+2
| | | | | | Sponsored by: Advanced Micro Devices (AMD) Sponsored by: The FreeBSD Foundation MFC after: 1 week
* iommu: disable dma by defaultEd Maste2024-08-131-1/+1
| | | | | | | | | | | | APIC ID 255 and above require x2APIC and DMAR interrupt remapping. FreeBSD is starting to be tested on high core count Intel systems that meet this criteria. We're going to enable DMAR by default to support this, so default hw.iommu.dma to 0 to avoid a significant performance regression. Reviewed by: kib, jhb Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D42435
* iommu_gas_remove(): consolidate places where decision is made to keep the entryKonstantin Belousov2023-12-261-2/+1
| | | | | | | Move all flag checks into iommu_gas_remove_unmap(). Sponsored by: The FreeBSD Foundation MFC after: 1 week
* iommu_gas: make placeholder entry at the start of the GAS zero sizeKonstantin Belousov2023-12-261-13/+6
| | | | | | | | | | same as the placeholder at the end. This is required to allow GAS to start at zero, for integration with vmm. Also, in iommu_gas_remove(), accept placeholders after remove op. Sponsored by: The FreeBSD Foundation MFC after: 1 week
* iommu: add iommu_gas_remove_locked()Konstantin Belousov2023-12-261-26/+57
| | | | | Sponsored by: The FreeBSD Foundation MFC after: 1 week
* IOMMU: add GAS map entry flag IOMMU_MAP_ENTRY_FAKEKonstantin Belousov2023-12-262-1/+6
| | | | | | | | to allow to shut down assert in iommu_gas_cmp_entries() when used against fake entry to search for specific place in the tree. Sponsored by: The FreeBSD Foundation MFC after: 1 week
* iommu_gas: zero fake on-stack map entry used in iommu_gas_remove_clip_left()Konstantin Belousov2023-12-261-0/+1
| | | | | Sponsored by: The FreeBSD Foundation MFC after: 1 week
* iommu_gas: print entries flags and domains if an overlap detectedKonstantin Belousov2023-12-261-3/+5
| | | | | Sponsored by: The FreeBSD Foundation MFC after: 1 week
* iommu_gas: add ddb 'show iommu_domain' commandKonstantin Belousov2023-12-261-0/+45
| | | | | Sponsored by: The FreeBSD Foundation MFC after: 1 week
* iommu_gas_match_one(): check for underflowKonstantin Belousov2023-12-261-0/+2
| | | | | Sponsored by: The FreeBSD Foundation MFC after: 1 week
* iommu: remove leftover sys/cdefs.h includesKonstantin Belousov2023-12-261-1/+0
| | | | | Sponsored by: The FreeBSD Foundation MFC after: 1 week
* busdma: remove parent tag trackingMitchell Horne2023-12-061-16/+7
| | | | | | | | | | Without filter functions, we do not need to keep track of tag ancestry. All inheritance of the parent tag's parameters occurs when creating the new child tag. Reviewed by: jhb Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D42895
* busdma: kill filter functionality internallyMitchell Horne2023-12-061-6/+5
| | | | | | | | | | | | | | Address filter functions are unused, unsupported, and now rejected. Simplify some busdma code by removing filter functionality completely. Note that the chains of parent tags become useless, and will be cleaned up in the next commit. No functional change intended. Reviewed by: jhb Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D42894
* sys: Automated cleanup of cdefs and other formattingWarner Losh2023-11-271-1/+0
| | | | | | | | | | | | | | | | Apply the following automated changes to try to eliminate no-longer-needed sys/cdefs.h includes as well as now-empty blank lines in a row. Remove /^#if.*\n#endif.*\n#include\s+<sys/cdefs.h>.*\n/ Remove /\n+#include\s+<sys/cdefs.h>.*\n+#if.*\n#endif.*\n+/ Remove /\n+#if.*\n#endif.*\n+/ Remove /^#if.*\n#endif.*\n/ Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/types.h>/ Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/param.h>/ Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/capsicum.h>/ Sponsored by: Netflix
* sys: Remove $FreeBSD$: one-line .c patternWarner Losh2023-08-162-4/+0
| | | | Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/
* sys: Remove $FreeBSD$: two-line .h patternWarner Losh2023-08-165-10/+0
| | | | Remove /^\s*\*\n \*\s+\$FreeBSD\$$\n/
* spdx: The BSD-2-Clause-FreeBSD identifier is obsolete, drop -FreeBSDWarner Losh2023-05-126-6/+6
| | | | | | | | | The SPDX folks have obsoleted the BSD-2-Clause-FreeBSD identifier. Catch up to that fact and revert to their recommended match of BSD-2-Clause. Discussed with: pfg MFC After: 3 days Sponsored by: Netflix
* iommu_gas: initialize start_gap as first nodeDoug Moore2023-02-081-1/+1
| | | | | | | | | | | | | | | | | | In iommu_gas.c, domain->start_gap points to one of the nodes on either side of the first free, unallocated range. In iommu_gas_init_domain, it is initialized to point to the node after the single free range. Change it to point to the node before that free range, so that, when 'lowaddr' is within the initial free range, the first allocation search for free space below 'lowaddr' does not begin and end at an address above 'lowaddr'. This fixes problems on a machine with Intel DMAR enabled. Reported by: jah Reviewed by: dougm Tested by: jah Obtained from: jah Fixes: commit db151ca0c343531256b8839f938a4ecbd8b4fd7e iommu_gas: start space search from 1st free space MFC after: 1 day
* iommu_gas: avoid overflow in bounds checkDoug Moore2022-12-221-8/+8
| | | | | | | | | | | Change the range test in iommu_gas_match_one from '< ubound' to '<= ubound', and pass a smaller-by-one ubound parameter to it, to avoid overflow in ubound calculation. Reported by: andrew Reviewed by: andrew (previous version) MFC after: 3 days Differential Revision: https://reviews.freebsd.org/D37764
* iommu_gas: drop fini cleanup loopDoug Moore2022-10-291-12/+4
| | | | | | | | | Assertions suggest that the loop in iommu_gas_fini_domain is executed zero times, so remove it. Reviewed by: alc, kib MFC after: 3 weeks Differential Revision: https://reviews.freebsd.org/D37204
* iommu_gas: start space search from 1st free spaceDoug Moore2022-10-292-19/+51
| | | | | | | | | | | | | | | | | | Maintain a pointer to an element in the domain map that is left of any sufficiently large free gap in the tree and start the search for free space there, rather than at the root of the tree. On find_space, move that pointer to the leftmost leaf in the subtree of nodes with free_down greater than or equal to the minimum allocation size before starting the search for space from that pointer. On removal of a node with address less than that pointer, update that pointer to point to the predecessor or successor of the removed node. In experiments with netperf streaming, this reduces by about 40% the number of map entries examined in first-fit allocation. Reviewed by: alc, kib MFC after: 3 weeks Differential Revision: https://reviews.freebsd.org/D36624
* rb_tree: let insert search start from next nodeDoug Moore2022-10-031-31/+29
| | | | | | | | | | | | | | | When the node to insert in the rb_tree is known to precede or follow a particular node, new methods RB_INSERT_PREV and RB_INSERT_NEXT, defined here, allow the search for where to insert the new node begin with that particular node, rather than at the root, to save a bit of time. Using those methods, instead of RB_INSERT, in managing a tree in iommu_gas.c, saves a little time. Reviewed by: kib MFC after: 3 weeks Differential Revision: https://reviews.freebsd.org/D35516
* kmem_malloc/free: Use void * instead of vm_offset_t for kernel pointers.John Baldwin2022-09-221-2/+2
| | | | | | Reviewed by: kib, markj Sponsored by: DARPA Differential Revision: https://reviews.freebsd.org/D36549
* rb_tree: augmentation shortcutDoug Moore2022-09-211-18/+42
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | RB-tree augmentation maintains data in each node of the tree that represents the product of some associative operator applied to all the nodes of the subtree rooted at that node. If a node in the tree changes, augmentation data for the node is updated for that node and all nodes on the path from that node to the tree root. However, sometimes, augmenting a node changes no data in that node, particularly if the associated operation is something involving 'max' or 'min'. If augmentation changes nothing in a node, then the work of walking to the tree root from that point is pointless, because augmentation will change nothing in those nodes either. This change makes it possible to avoid that wasted work. Define RB_AUGMENT_CHECK as a macro much like RB_AUGMENT, but which returns a value 'true' when augmentation changes the augmentation data of a node, and false otherwise. Change code that unconditionally walks and augments to the top of tree to code that stops once an augmentation has no effect. In the case of rebalancing the tree after insertion or deletion, where previously a node rotated into the path was inevitably augmented on the march to the tree root, now check to see if it needs augmentation because the march to the tree root stopped before reaching it. Change the augmentation function in iommu_gas.c so that it returns true/false to indicate whether the augmentation had any effect. Reviewed by: alc, kib MFC after: 3 weeks Differential Revision: https://reviews.freebsd.org/D36509
* iommu_gas: don't remove and insert boundary entryDoug Moore2022-09-161-10/+4
| | | | | | | | | | Avoid removing an item in iommu_gas_free_region only to reinsert it, by avoiding removing an entry that is either first_place or last_place. Reviewed by: kib MFC after: 3 weeks Differential Revision: https://reviews.freebsd.org/D36597
* iommu_gas: add iommu_gas_remove()Konstantin Belousov2022-08-153-0/+122
| | | | | | | | | | | | | | | | The function removes the range of addresses from GAS. Right now it is unused. The initial version of the patch was extracted from the stalled work for integration of DMAR and bhyve, see D25672. Having the function in the main tree would allow it to co-evolve with other active changes to the IOMMU driver. Requested by: alc Reviewed by: alc, dougm Sponsored by: The FreeBSD Foundation MFC after: 1 week Differential revision: https://reviews.freebsd.org/D36001
* iommu_gas: Eliminate redundant parameters and push down lock acquisitionAlan Cox2022-07-303-27/+30
| | | | | | | | | | | | | | | | | | Since IOMMU map entries store a reference to the domain in which they reside, there is no need to pass the domain to iommu_gas_free_entry(), iommu_gas_free_space(), and iommu_gas_free_region(). Push down the acquisition and release of the IOMMU domain lock into iommu_gas_free_space() and iommu_gas_free_region(). Both of these changes allow for simplifications in the callers of the functions without really complicating the functions themselves. Moreover, the latter change eliminates the direct use of the IOMMU domain lock from the x86-specific DMAR code. Reviewed by: kib MFC after: 1 week Differential Revision: https://reviews.freebsd.org/D35995
* x86/iommu: Reduce DMAR lock contentionAlan Cox2022-07-292-3/+7
| | | | | | | | | | Replace the DMAR unit's tlb_flush TAILQ by a custom list implementation that enables dmar_qi_task() to dequeue entries without holding the DMAR lock. Reviewed by: kib MFC after: 1 week Differential Revision: https://reviews.freebsd.org/D35951
* iommu_gas: Eliminate a possible case of use-after-freeAlan Cox2022-07-253-4/+6
| | | | | | | | | | | | | Eliminate a possible case of use-after-free in an error handling path after a mapping failure. Specifically, eliminate IOMMU_MAP_ENTRY_QI_NF and instead perform the IOTLB invalidation synchronously. Otherwise, when iommu_domain_unload_entry() is called and told not to free the IOMMU map entry, the caller could free the entry before dmar_qi_task() is finished with it. Reviewed by: kib MFC after: 1 week Differential Revision: https://reviews.freebsd.org/D35878
* iommu_gas: Avoid double unmapping on errorAlan Cox2022-07-211-1/+1
| | | | | | | | | | | | | In the extremely unlikely case that the iommu_gas_map_region() call in bus_dma_iommu_load_ident() failed, we would attempt to unmap the failed entry twice, first in iommu_gas_map_region(), and a second time in the caller. Once is enough, and twice is problematic because it leads to a second RB_REMOVE call on the same tree node. Like it or not, RB_TREE does not handle that possibility. Reviewed by: kib MFC after: 1 week Differential Revision: https://reviews.freebsd.org/D35869
* iommu: Shrink the iommu map entry structureAlan Cox2022-07-162-25/+10
| | | | | | | | | Eliminate the unroll_entry field from struct iommu_map_entry, shrinking the struct by 16 bytes on 64-bit architectures. Reviewed by: kib MFC after: 2 weeks Differential Revision: https://reviews.freebsd.org/D35769
* iommu_gas: consolidate find_space helpersDoug Moore2022-07-101-159/+137
| | | | | | | | | | | Merge lowermatch and uppermatch into find_space. Eliminate uppermatch recursion. Merge match_insert into match_one and eliminate some redundant calculation. Move some initialization out of find_space and into map (and out from under a lock). Reviewed by: kib (previous version), alc MFC after: 3 weeks Differential Revision: https://reviews.freebsd.org/D35440
* iommu_gas: Fix a recent regression with IOMMU_MF_CANSPLITAlan Cox2022-06-261-1/+1
| | | | | | | | | | As of 19bb5a7244ff, the IOMMU_MF_CANSPLIT case in iommu_gas_match_one() must take into account the specified offset. Otherwise, the recently changed end calculation in iommu_gas_match_insert() could produce an end address that crosses the specified boundary by one page. Reviewed by: dougm MFC with: 19bb5a7244ff
* iommu_gas: Tidy upAlan Cox2022-06-261-13/+11
| | | | | | | | Move a comment to the code that it describes. Improve the wording. Style fixes. MFC after: 2 weeks
* busdma_iommu: Fine-grained locking for the dmamap's map listAlan Cox2022-06-252-29/+34
| | | | | | | | | | Introduce fine-grained locking on the dmamap's list of map entries, replacing the use of the domain lock. This is not the most significant source of lock contention, but it is the easiest to address. Reviewed by: kib MFC after: 2 weeks Differential Revision: https://reviews.freebsd.org/D35557
* iommu_gas: Add a missing error-case unlockAlan Cox2022-06-211-1/+3
| | | | | | | Release the domain lock when iommu_gas_reserve_region_extend()'s call to iommu_gas_reserve_region_locked() fails. MFC after: 2 weeks
* busdma_iommu: Eliminate a redundant trunc_page()Alan Cox2022-06-211-1/+1
| | | | | | | Since OFF_TO_IDX() inherently truncates the given value, there is no need to perform trunc_page() on it. MFC after: 2 weeks
* iommu_gas: Drop needless bound check in lowermatchDoug Moore2022-06-211-10/+5
| | | | | | | | | | | | The loop iteration in iommu_gas_lowermatch checks the bound a->common->lowaddr twice per loop iteration. Rewrite to test only once per iteration. Do not worry about passing to iommu_gas_match_one a range wholly beyond lowaddr. Since that function checks the upper end of the range against lowaddr, it'll get rejected there. Reviewed by: alc MFC after: 3 weeks Differential Revision: https://reviews.freebsd.org/D35522
generated by cgit v1.3 (git 2.53.0) at 2026-04-08 10:40:17 +0000