aboutsummaryrefslogtreecommitdiff
path: root/sys/arm64/include
Commit message (Collapse)AuthorAgeFilesLines
* arm64: Add a function to check a range of CPU revsAndrew Turner2026-01-141-0/+25
| | | | | | | | | | | Add a function that can check if a given midr is within a range of revisions. This will be used to check if a CPU is affected by a known erratum. Sponsored by: Arm Ltd Differential Revision: https://reviews.freebsd.org/D52187 (cherry picked from commit c76b0247a95ed090cc0d83b2698228d2937af3e6)
* arm64: Replace cpu_tlb_flushID in initarmAndrew Turner2026-01-141-0/+2
| | | | | | | | | | | Replace cpu_tlb_flushID with pmap_s1_invalidate_all_kernel in initarm. They are identical in the SMP case. Reviewed by: alc Sponsored by: Arm Ltd Differential Revision: https://reviews.freebsd.org/D52184 (cherry picked from commit fa23bfb425d53a87064d86ec9ae0ae5e31aa5d5b)
* arm64: Read the CPU feature tunables onceAndrew Turner2026-01-141-0/+3
| | | | | | | | | | | | | Rather than reading the CPU tunables each time we enable a feature just read them once on boot. Reading them early on non-boot CPUs causes issues where they try to allocate memory before the core is set up to do so. As they don't change it is safe to cache the state of them. Sponsored by: Arm Ltd Differential Revision: https://reviews.freebsd.org/D52578 (cherry picked from commit 5e0e2565288f3d1f1b4223d8bb53c7d70e0aa92c)
* arm64: Add cpu_feat_disabled for disabled featuresAndrew Turner2026-01-141-1/+4
| | | | | | | | | | | When a feature is disabled we may need to run a cleanup handler, e.g. to remove a feature from the sanitized ID registers. Add support for this with a new feat_disabled handler. Sponsored by: Arm Ltd Differential Revision: https://reviews.freebsd.org/D52577 (cherry picked from commit 055229eda697445880edd0050d0230a3f1bc85b3)
* arm64: Support managing features from loaderAndrew Turner2026-01-141-1/+26
| | | | | | | | | | | | | Allow the user to enable/disable supported cpu features/errata from a known tunable. This will allow the user to disable features, e.g. to work around broken firmware. It can also be used to enable workarounds for rare errata. Reviewed by: imp Sponsored by: Arm Ltd Differential Revision: https://reviews.freebsd.org/D52358 (cherry picked from commit 4bc68fa98f68211ee7943d77cfc91f60ccb5880d)
* arm64: Add a sysctl to see if features are enabledAndrew Turner2026-01-141-3/+10
| | | | | | | | | | | This will also be used as a tunable to control features in a later change. Reviewed by: imp Sponsored by: Arm Ltd Differential Revision: https://reviews.freebsd.org/D52357 (cherry picked from commit 9204a315a71c5aa9a9b8c11f2dcefb155dd5fc34)
* arm64: Add a macro to create cpu_featAndrew Turner2026-01-141-0/+10
| | | | | | | | | | This will be used later to extend cpu feature detection Reviewed by: imp (earlier version) Sponsored by: Arm Ltd Differential Revision: https://reviews.freebsd.org/D52356 (cherry picked from commit b3605a7bb6f8c58d939e642671ce70b0ce39b95e)
* arm64: Add a new CPU feature frameworkAndrew Turner2026-01-141-0/+88
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This will be used to enable features that are discoverable on boot. It has support to check if a feature needs to be enabled, and if there is any errata associated with the feature that mean it shouldn't be enabled. It can also be used to enable errata workarounds that aren't associated with a specific feature, e.g. where we need to trap access to a register in userspace to emulate it. As some features need to be enabled early, while others can wait there are two options for when to enable a feature: * CPU_FEAT_EARLY_BOOT: The feature will be enabled early in the boot. On the boot CPU this is in initarm, while on secondary CPUs this is before signalling the CPU has started. * CPU_FEAT_AFTER_DEV: The feature will be enabled after devices have attached but before interrupts are enabled. There are also two scopes for where to enable the feature: * CPU_FEAT_PER_CPU: The feature will be checked and enabled on all CPU cores. * CPU_FEAT_SYSTEM: The feature will only be checked and enabled on a single core. It is expected the former will be used for most features, while the latter is for features that set a global variable to be checked. Sponsored by: Arm Ltd Differential Revision: https://reviews.freebsd.org/D47812 (cherry picked from commit ac4fa5838bb33f0c3ba05fce02d41164bd84a560)
* arm64: Add the new C1 CPU IDsAndrew Turner2026-01-141-0/+4
| | | | | | | | | Add the Arm C1-Nano, C1-Pro, C1-Premium, and C1-Ultra CPUs from their Technical Reference Manuals. Sponsored by: Arm Ltd (cherry picked from commit 8fee6b9ecc84d3602a461f1cd33df91e50849cdf)
* arm64: Add more CPU MIDR valuesAndrew Turner2026-01-141-1/+19
| | | | | | | | | Found in Linux and https://github.com/arm-software/data Sponsored by: Arm Ltd Differential Revision: https://reviews.freebsd.org/D50726 (cherry picked from commit 124b5dbf5c09a17251b75f6b96c9ab7b218eee7f)
* runq: Deduce most parameters, remove machine headersOlivier Certner2025-07-281-50/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The 'runq' machinery now depends on only two settable parameters, RQ_MAX_PRIO, the maximum priority number that can be accepted, the minimum being 0, and RQ_PPQ, the number of priorities per queue (to reduce the number of queues). All other parameters are deduced from these ones. Also, all architectures automatically get a runq word that is their natural word. RQB_FFS() always was 'ffsl() - 1' except for amd64 where it was 'bsfq()'. Now that all these finally call compiler builtins, the resulting assembly code is the same, so there is no cost to removing this special case. After all these changes, <machine/runq.h> headers have no more purpose, so remove them. While here, fix potentially confusing parameter name for RQB_WORD() and RQB_BIT(). While here, include all necessary headers so that <sys/runq.h> can be included standalone. No functional change (intended). Reviewed by: kib MFC after: 1 month Event: Kitchener-Waterloo Hackathon 202506 Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D45387 (cherry picked from commit 2fefe2c88b31dc7f173c9424f3eb13d49e98d55a)
* Don't rely on sys/_types.h including sys/cdefs.hBrooks Davis2025-06-231-0/+2
| | | | | | | | | | | | | | These headers relied in __BEGIN_DECS/__END_DECLS being defined when sys/_types.h was included, but there's not a requirement that this be the case. Reviewed by: imp Exp-run by: antoine (PR 286274) Pull Request: https://github.com/freebsd/freebsd-src/pull/1595 (cherry picked from commit b01e971fd39d10273a387788019457792873e81c) (cherry picked from commit 4f4c465b45d42ed8d2b65c8b4c909246987ac877) (cherry picked from commit 12913e4ddf710e517f3ae00b7b420b783cca9b37)
* arm64: bus: Add 36-bit address mask for use in bus space allocationsKa Ho Ng2025-03-151-0/+1
| | | | | | | | | Obtained from: Juniper Networks, Inc. MFC after: 1 week Reviewed by: imp Differential Revision: https://reviews.freebsd.org/D44764 (cherry picked from commit 5484c3d6eb31a78f07ed5e66862a1e7d3b5225b6)
* atomics: Constify loadsOlivier Certner2025-01-161-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | In order to match reality, allow using these functions with pointers on const objects, and bring us closer to C11. Remove the '+' modifier in the atomic_load_acq_64_i586()'s inline asm statement's constraint for '*p' (the value to load). CMPXCHG8B always writes back some value, even when the value exchange does not happen in which case what was read is written back. atomic_load_acq_64_i586() further takes care of the operation atomically writing back the same value that was read in any case. All in all, this makes the inline asm's write back undetectable by any other code, whether executing on other CPUs or code on the same CPU before and after the call to atomic_load_acq_64_i586(), except for the fact that CMPXCHG8B will trigger a #GP(0) if the memory address is part of a read-only mapping. This unfortunate property is however out of scope of the C abstract machine, and in particular independent of whether the 'uint64_t' pointed to is declared 'const' or not. Approved by: markj (mentor) MFC after: 5 days Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D46887 (cherry picked from commit 5e9a82e898d55816c366cfa3ffbca84f02569fe5)
* atomic(9): Implement atomic_testand(clear|set)_ptrJohn Baldwin2024-12-011-0/+2
| | | | | | | | | | | | For current architectures, these are just aliases for the existing operation on the relevant scalar integer. Reviewed by: imp, kib Obtained from: CheriBSD Sponsored by: AFRL, DARPA Differential Revision: https://reviews.freebsd.org/D47631 (cherry picked from commit a80b9ee15aa0f2da9f9f0c48e13324e0e5f2e840)
* arm64: Stop trashing x28 in savectxAndrew Turner2024-10-211-1/+1
| | | | | | | | | While here make it return void, we don't set any useful return value and nothing checks for it. Sponsored by: Arm Ltd (cherry picked from commit a7c5c88cb29ff512467a6599f1b4d8e2b8817480)
* arm64: Enable SVE in userspaceAndrew Turner2024-10-211-0/+1
| | | | | | | | | | | Report when SVE is present and allow it to be used by calling sve_restore_state on an SVE exception from userspace. Reviewed by: kib Sponsored by: Arm Ltd Differential Revision: https://reviews.freebsd.org/D43310 (cherry picked from commit 205c1007be6d57b2eb8620f79384cb2299dde40e)
* arm64: Don't trap SVE to EL2Andrew Turner2024-10-211-1/+1
| | | | | | | | | | | As with floating point instructions don't trap SVE instructions to the hypervisor. This lets us handle then in the kernel. Reviewed by: imp (earlier version) Sponsored by: Arm Ltd Differential Revision: https://reviews.freebsd.org/D43303 (cherry picked from commit fe5ed2496e44aec018a6215175bba225b20d81fd)
* arm64: Support SVE in ptrace and core dumpsAndrew Turner2024-10-211-0/+13
| | | | | | | | | | | | | | Add the NT_ARM_SVE note type and use it to access the SVE registers from ptrace. This allows userspace to modify the full SVE register values. Try to follow the Linux semantics to allow debuggers to use this with minimal changes. Sponsored by: Arm Ltd Differential Revision: https://reviews.freebsd.org/D43309 (cherry picked from commit aab60068943d733b0b4573e5481c543ab3d45a00)
* arm64: Add an SVE sysarchAndrew Turner2024-10-211-0/+3
| | | | | | | | | | | To allow for user space to read the SVE vector length add a sysarch handler to return the value to userspace. Reviewed by: imp Sponsored by: Arm Ltd Differential Revision: https://reviews.freebsd.org/D43308 (cherry picked from commit 87a58d931db7b40d8d225a7edb82b78937078b7f)
* arm64: Add the SVE registers to the signal frameAndrew Turner2024-10-211-0/+8
| | | | | | | | | | | | | | Use the new extended register support in the arm64 signal frame to handle the SVE registers. As these registers alias the VFP registers we use the floating point register values to restore the lower bits of the SVE registers. This is to support software that doesn't understand SVE to continue working. Sponsored by: Arm Ltd Differential Revision: https://reviews.freebsd.org/D43307 (cherry picked from commit e45132e978030f0aa964d975a9fbad5614f260bd)
* arm64: Initial SVE supportAndrew Turner2024-10-213-3/+18
| | | | | | | | | | | | | | | | Add initial kernel support for SVE. This detects if SVE is present on all CPUs, and if so allows for the use of SVE in the future. As the SVE registers are a superset of the VFP registers we don't need to restore the VFP registers when SVE is enabled. Ths interface to enable SVE is provided, but not used until SVE is supported in signals and with ptrace. Sponsored by: Arm Ltd Differential Revision: https://reviews.freebsd.org/D43306 (cherry picked from commit 332c426328dbb30a6b2e69d9b1e8298d77d85bd1)
* arm64: Create a version of vfp_save_state for cpu_switchAndrew Turner2024-10-211-0/+1
| | | | | | | | | | | This will be used when we add SVE support to reduce the registers needed to be saved on context switch. Reviewed by: imp Sponsored by: Arm Ltd Differential Revision: https://reviews.freebsd.org/D43305 (cherry picked from commit a30149b2a9c6ac5280523eea9570e5b5e5f1fdf8)
* arm64: Split out a savectx version of vfp_save_stateAndrew Turner2024-10-211-0/+1
| | | | | | | | | | | Rather than try to detect when vfp_save_state is called by savectx use a separate function that sets up the pcb as needed. Reviewed by: imp Sponsored by: Arm Ltd Differential Revision: https://reviews.freebsd.org/D43304 (cherry picked from commit 12257233e8fd94ab24e1a84ad87126af2a7be33b)
* arm64: Adjust the indentation of CPTR_EL2 valuesAndrew Turner2024-10-211-6/+6
| | | | | | | | Reviewed by: emaste Sponsored by: Arm Ltd Differential Revision: https://reviews.freebsd.org/D46513 (cherry picked from commit 7a488d83b3af4d59946319b251a3a2060f18df40)
* arm64: add additional MDCR_EL2 fieldsZachary Leaf2024-10-211-0/+30
| | | | | | | | | | | Monitor Debug Configuration Register provides EL2 configuration options for self-hosted debug and the Performance Monitors Extension. Reviewed by: andrew Sponsored by: Arm Ltd Differential Revision: https://reviews.freebsd.org/D46191 (cherry picked from commit 610348a90467980de0498fab8dfdddf221d7a604)
* arm64: Add the tcr_el2 ds fieldAndrew Turner2024-10-211-0/+2
| | | | | | | | | | | This will be used to support FEAT_LPA2 to allow more than 48 bits of physical address space. Reviewed by: alc, kib, markj Sponsored by: Arm Ltd Differential Revision: https://reviews.freebsd.org/D46392 (cherry picked from commit 7be11454edd4eca6cbd299cb5b486294f912ffbd)
* arm64: Fix the ESR_EL1_op2 valueAndrew Turner2024-10-211-1/+1
| | | | | | | | | This leads to an unallocated register. Fix the value to point to ESR_EL1. Sponsored by: Arm Ltd (cherry picked from commit 8b017284319c46d349abee5573d4656bb377b0fd)
* arm64: Add EL1 and EL12 register alt namesAndrew Turner2024-10-211-0/+222
| | | | | | | | | | | | | When adding VHE support we may need to access these registers using the _EL12 or _EL1 name depending on if VHE is enabled or not. Add the registers we will likely need to access from VHE so these can be updated in the guest switcher code. Sponsored by: Arm Ltd Differential Revision: https://reviews.freebsd.org/D45525 (cherry picked from commit 47361851bc2e292bd7c723ba5f57792a7794f1b4)
* arm64: add CONTEXTIDR_EL1 regZachary Leaf2024-10-211-0/+9
| | | | | | | | | | | CONTEXTIDR_EL1 is used in debug and trace features to identify the current process or context. Reviewed by: andrew Sponsored by: Arm Ltd Differential Revision: https://reviews.freebsd.org/D45173 (cherry picked from commit 4f8ba1c9dd00e0ae20fa161e166715746b6d1c04)
* armv8rng: Don't require toolchain to support FEAT_RNGJessica Clarke2024-10-211-0/+9
| | | | | | | | | We have the mechanism in place to support encoding system registers explicitly, so use that rather than requiring LLVM 13+, which breaks our current set of GitHub CI builds. Fixes: 9eecef052155 ("Add an Armv8 rndr random number provider") (cherry picked from commit 9560ac4b638edf688566f576adc65d3654f2240c)
* intrng: Extract arm/arm64 IPI->PIC glue codeJessica Clarke2024-09-061-4/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | The arm and arm64 implementations of dispatching IPIs via PIC_IPI_SEND are almost identical, and entirely MI with the lone exception of a single store barrier on arm64 (that is likely either redundant or needed on arm too). Thus, de-duplicate this code by moving it to INTRNG as a generic IPI glue framework. The ipi_* functions remain declared in MD smp.h headers and implemented in MD code, but are trivial wrappers around intr_ipi_send that could be made MI, at least for INTRNG ports, at a later date. Note that, whilst both arm and arm64 had an ii_send member in intr_ipi to abstract over how to send interrupts,, they were always ultimately using PIC_IPI_SEND, and so this complexity has been removed. A follow-up commit will re-introduce the same flexibility by instead allowing a device other than the root PIC to be registered as the IPI sender. As part of this, strengthen a MAXCPU assertion that was missed in commit 2f0b059eeafc ("intrng: switch from MAXCPU to mp_ncpus") (which itself is mis-titled). Reviewed by: mmel, mhorne MFC after: 1 month Differential Revision: https://reviews.freebsd.org/D35898 (cherry picked from commit fae8755f16ff5b9bdc32df046e0f16c0cbb48a29)
* arm64: Expand the use of Armv8.1-A atomicsAndrew Turner2024-09-021-2/+3
| | | | | | | | | | | | When targeting Armv8.1 we can assume FEAT_LSE is available and can use the atomic instructions this provides without needing to check for support first. Reviewed by: imp, markj, emaste Sponsored by: Arm Ltd Differential Revision: https://reviews.freebsd.org/D46159 (cherry picked from commit 7a345763f96d86eee9ab578e64311bf452e58900)
* arm64: Ensure sctlr and pstate are in known statesAndrew Turner2024-09-021-0/+4
| | | | | | | | | | | | | | | | Before entering the kernel exception level ensure sctlr_el2 and sctlr_el1 are in a known state. The EOS flag needs to be set to ensure an eret instruction is a context synchronization event. Set spcr_el1 when entering the kernel from EL1 and use an eret instruction to return to the caller. This ensures the CPU pstate is consistent with the value in spcr_el1 as it is the only way to set it directly. Sponsored by: Arm Ltd Differential Revision: https://reviews.freebsd.org/D45528 (cherry picked from commit 034c83fd7d85f57193850a73cc0ac957a211f725)
* arm64: Support counter access with E2HAndrew Turner2024-09-021-2/+7
| | | | | | | | | | | When entering the kernel with the E2H field set the layout of the cnthctl_el2 register changes. Use the correct field locations to enable access to the counter and timer registers from EL1. Sponsored by: Arm Ltd Differential Revision: https://reviews.freebsd.org/D45529 (cherry picked from commit 997511dffe651e1d2d708f37f2ced430a6ab3349)
* arm64: Support passing more registers to signalsAndrew Turner2024-09-021-1/+15
| | | | | | | | | | | | | | | To support recent extensions to the Arm architecture we may need to store more or larger registers when sending a signal. To support this create a list of these extra registers. Userspace that needs to access a register in the signal handler can then walk the list to find the correct register struct and read/write its contents. Reviewed by: kib, markj (earlier version) Sponsored by: Arm Ltd Differential Revision: https://reviews.freebsd.org/D43302 (cherry picked from commit 7e6437c08415ade75403f1ecad75167257e8ea3c)
* arm64: Fix indentation to be consistentAndrew Turner2024-07-151-7/+7
| | | | | | | | | | | Adjust the mair_el1 macro indentation to be consistent with the surrounding macros. Reviewed by: emaste Sponsored by: Arm Ltd Differential Revision: https://reviews.freebsd.org/D45524 (cherry picked from commit 86bafddd61aba115bc46bcf1d7e0afb125850b5f)
* arm64: Add the pointer auth registers to armreg.hAndrew Turner2024-07-151-0/+80
| | | | | | | | | | | Add the pointer authentication registers to armreg.h. These will be used to support pointer authentication in a kernel built with GCC. Reviewed by: jhb Sponsored by: Arm Ltd Differential Revision: https://reviews.freebsd.org/D45262 (cherry picked from commit 57d714a23f5ce21e389d53636b8bc6c1b45d518e)
* arm64: Use the UL macro in TCR_EL1 definesAndrew Turner2024-07-151-48/+48
| | | | | | | | | | | | While clang can handle numbers with a UL suffix in assembly files gcc/gas is unable to. Switch to use the UL macro for TCR_EL1 defines as some are used in locore.S Reviewed by: brooks, jhb Sponsored by: Arm Ltd Differential Revision: https://reviews.freebsd.org/D45261 (cherry picked from commit 29c1cf9860e531146220d9dc3596e4c79f91cfcd)
* arm64: add PMBSR_MSS_{BSC,FSC} status code fieldZachary Leaf2024-07-151-0/+2
| | | | | | | | | | | | | Bits [5:0] of PMBSR_MSS encodes either Buffer Status Code (BSC) or Fault Status Code (FSC) depending on PMBSR_EC value. Add PMBSR_MSS_{BSC,FSC} to cover this field. Reviewed by: andrew Sponsored by: Arm Ltd Differential Revision: https://reviews.freebsd.org/D45172 (cherry picked from commit 10b3eac88db689d657c4d0d0716bcbdf240ff614)
* arm64: make SPE regs use ALT_NAME macroZachary Leaf2024-07-151-60/+72
| | | | | | | | | | | | | | | | | When the register is not defined in Armv8.0 i.e. added in a later extension, like SPE added in v8.2, the alternative name format of: S<op0>_<op1>_C<crn>_C<crm>_<op2> should be used; otherwise, calls to {READ,WRITE}_SPECIALREG() will fail. Use the MRS_REG_ALT_NAME() macro for SPE changing hex to decimal as required by the macro. Reviewed by: andrew Sponsored by: Arm Ltd Differential Revision: https://reviews.freebsd.org/D45171 (cherry picked from commit f7bdaa103eb8906fc999c7fd5e8d6af440e26e6c)
* arm64: Add MRS_REG_ALT_NAME ID register macrosAndrew Turner2024-07-151-0/+15
| | | | | | | | | | | These can be used even when the compiler is too old for the register to be included. Reviewed by: Zachary Leaf <zachary.leaf@arm.com> Sponsored by: Arm Ltd Differential Revision: https://reviews.freebsd.org/D45176 (cherry picked from commit d6d860c7ff5c4cbe9475d98000407d6f0ea84b47)
* am64: Allow cpu.h to be included from assemblyAndrew Turner2024-07-151-0/+4
| | | | | | | | Reviewed by: jhibbits, kevans Sponsored by: Arm Ltd Differential Revision: https://reviews.freebsd.org/D45081 (cherry picked from commit e353ac0cfd5d155c01253ee17c2bf23f888cb7de)
* arm64: Add EL1 hardware breakpoint exceptionsAndrew Turner2024-07-152-1/+4
| | | | | | | | Reviewed by: jhb Sponsored by: Arm Ltd Differential Revision: https://reviews.freebsd.org/D44353 (cherry picked from commit c802b486ddfd796ea35b770148af9a5b0cd0ee64)
* arm64: Add a macro to find a VM fault addressAndrew Turner2024-07-151-0/+2
| | | | | | | | | | | Add a macro to find which bits from far_el2 are needed to be copied to get the full intermediate physical address (IPA). The hpfar_el2 register only contains a 4k aligned fault address. We need to include the lower bits from far_el2 if we need the full faulting IPA. (cherry picked from commit b3bbec37ba039a46f7166f6fa3571f38da4253e9)
* arm64: Add the TCR_EL2.PS maskAndrew Turner2024-07-151-0/+1
| | | | (cherry picked from commit 9c52f98c9f1abfe6577335522b6007659f759adc)
* arm64: Add ISS_MSR_REG for ESR_ELx.ISS valuesAndrew Turner2024-07-151-0/+6
| | | | | | | Add a macro to get the ESR_ELx ISS value when we trap accessing a special register. (cherry picked from commit 09ac9cf8971a0709bb8d5a3a703cd3dbff882b6f)
* arm64: Add more spsr_el1 register valuesAndrew Turner2024-07-151-0/+7
| | | | | | Sponsored by: Arm Ltd (cherry picked from commit 64963dd2f0a2a47e94fb3f7c56198c475b9821a3)
* arm64: Add CurrentEL register definitionsAndrew Turner2024-07-151-0/+8
| | | | | | | Sponsored by: Arm Ltd Differential Revision: https://reviews.freebsd.org/D43971 (cherry picked from commit 5e7941b6c979a55dc67baf316db6932c09b14022)
* arm64, riscv: removed unused struct pv_addrMitchell Horne2024-05-271-10/+0
| | | | | | | | | | | No functional change. Reviewed by: markj MFC after: 3 days Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D45322 (cherry picked from commit b5e17840de65ff196b2398211b68ce96d6919f2c)