| Commit message (Collapse) | Author | Age | Files | Lines |
| |
|
|
|
|
|
|
|
|
|
| |
Add a function that can check if a given midr is within a range of
revisions. This will be used to check if a CPU is affected by a known
erratum.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D52187
(cherry picked from commit c76b0247a95ed090cc0d83b2698228d2937af3e6)
|
| |
|
|
|
|
|
|
|
|
|
| |
Replace cpu_tlb_flushID with pmap_s1_invalidate_all_kernel in initarm.
They are identical in the SMP case.
Reviewed by: alc
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D52184
(cherry picked from commit fa23bfb425d53a87064d86ec9ae0ae5e31aa5d5b)
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
Rather than reading the CPU tunables each time we enable a feature
just read them once on boot. Reading them early on non-boot CPUs
causes issues where they try to allocate memory before the core is set
up to do so. As they don't change it is safe to cache the state of
them.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D52578
(cherry picked from commit 5e0e2565288f3d1f1b4223d8bb53c7d70e0aa92c)
|
| |
|
|
|
|
|
|
|
|
|
| |
When a feature is disabled we may need to run a cleanup handler, e.g.
to remove a feature from the sanitized ID registers. Add support for
this with a new feat_disabled handler.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D52577
(cherry picked from commit 055229eda697445880edd0050d0230a3f1bc85b3)
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
Allow the user to enable/disable supported cpu features/errata from a
known tunable. This will allow the user to disable features, e.g. to
work around broken firmware. It can also be used to enable workarounds
for rare errata.
Reviewed by: imp
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D52358
(cherry picked from commit 4bc68fa98f68211ee7943d77cfc91f60ccb5880d)
|
| |
|
|
|
|
|
|
|
|
|
| |
This will also be used as a tunable to control features in a later
change.
Reviewed by: imp
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D52357
(cherry picked from commit 9204a315a71c5aa9a9b8c11f2dcefb155dd5fc34)
|
| |
|
|
|
|
|
|
|
|
| |
This will be used later to extend cpu feature detection
Reviewed by: imp (earlier version)
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D52356
(cherry picked from commit b3605a7bb6f8c58d939e642671ce70b0ce39b95e)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This will be used to enable features that are discoverable on boot. It
has support to check if a feature needs to be enabled, and if there is
any errata associated with the feature that mean it shouldn't be
enabled. It can also be used to enable errata workarounds that aren't
associated with a specific feature, e.g. where we need to trap access
to a register in userspace to emulate it.
As some features need to be enabled early, while others can wait there
are two options for when to enable a feature:
* CPU_FEAT_EARLY_BOOT: The feature will be enabled early in the boot.
On the boot CPU this is in initarm, while on secondary CPUs this
is before signalling the CPU has started.
* CPU_FEAT_AFTER_DEV: The feature will be enabled after devices have
attached but before interrupts are enabled.
There are also two scopes for where to enable the feature:
* CPU_FEAT_PER_CPU: The feature will be checked and enabled on all
CPU cores.
* CPU_FEAT_SYSTEM: The feature will only be checked and enabled on a
single core.
It is expected the former will be used for most features, while the
latter is for features that set a global variable to be checked.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D47812
(cherry picked from commit ac4fa5838bb33f0c3ba05fce02d41164bd84a560)
|
| |
|
|
|
|
|
|
|
| |
Add the Arm C1-Nano, C1-Pro, C1-Premium, and C1-Ultra CPUs from their
Technical Reference Manuals.
Sponsored by: Arm Ltd
(cherry picked from commit 8fee6b9ecc84d3602a461f1cd33df91e50849cdf)
|
| |
|
|
|
|
|
|
|
| |
Found in Linux and https://github.com/arm-software/data
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D50726
(cherry picked from commit 124b5dbf5c09a17251b75f6b96c9ab7b218eee7f)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The 'runq' machinery now depends on only two settable parameters,
RQ_MAX_PRIO, the maximum priority number that can be accepted, the
minimum being 0, and RQ_PPQ, the number of priorities per queue (to
reduce the number of queues).
All other parameters are deduced from these ones. Also, all
architectures automatically get a runq word that is their natural word.
RQB_FFS() always was 'ffsl() - 1' except for amd64 where it was
'bsfq()'. Now that all these finally call compiler builtins, the
resulting assembly code is the same, so there is no cost to removing
this special case.
After all these changes, <machine/runq.h> headers have no more purpose,
so remove them.
While here, fix potentially confusing parameter name for RQB_WORD() and
RQB_BIT().
While here, include all necessary headers so that <sys/runq.h> can be
included standalone.
No functional change (intended).
Reviewed by: kib
MFC after: 1 month
Event: Kitchener-Waterloo Hackathon 202506
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D45387
(cherry picked from commit 2fefe2c88b31dc7f173c9424f3eb13d49e98d55a)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
These headers relied in __BEGIN_DECS/__END_DECLS being defined when
sys/_types.h was included, but there's not a requirement that this be
the case.
Reviewed by: imp
Exp-run by: antoine (PR 286274)
Pull Request: https://github.com/freebsd/freebsd-src/pull/1595
(cherry picked from commit b01e971fd39d10273a387788019457792873e81c)
(cherry picked from commit 4f4c465b45d42ed8d2b65c8b4c909246987ac877)
(cherry picked from commit 12913e4ddf710e517f3ae00b7b420b783cca9b37)
|
| |
|
|
|
|
|
|
|
| |
Obtained from: Juniper Networks, Inc.
MFC after: 1 week
Reviewed by: imp
Differential Revision: https://reviews.freebsd.org/D44764
(cherry picked from commit 5484c3d6eb31a78f07ed5e66862a1e7d3b5225b6)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
In order to match reality, allow using these functions with pointers on
const objects, and bring us closer to C11.
Remove the '+' modifier in the atomic_load_acq_64_i586()'s inline asm
statement's constraint for '*p' (the value to load). CMPXCHG8B always
writes back some value, even when the value exchange does not happen in
which case what was read is written back. atomic_load_acq_64_i586()
further takes care of the operation atomically writing back the same
value that was read in any case. All in all, this makes the inline
asm's write back undetectable by any other code, whether executing on
other CPUs or code on the same CPU before and after the call to
atomic_load_acq_64_i586(), except for the fact that CMPXCHG8B will
trigger a #GP(0) if the memory address is part of a read-only mapping.
This unfortunate property is however out of scope of the C abstract
machine, and in particular independent of whether the 'uint64_t' pointed
to is declared 'const' or not.
Approved by: markj (mentor)
MFC after: 5 days
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D46887
(cherry picked from commit 5e9a82e898d55816c366cfa3ffbca84f02569fe5)
|
| |
|
|
|
|
|
|
|
|
|
|
| |
For current architectures, these are just aliases for the existing
operation on the relevant scalar integer.
Reviewed by: imp, kib
Obtained from: CheriBSD
Sponsored by: AFRL, DARPA
Differential Revision: https://reviews.freebsd.org/D47631
(cherry picked from commit a80b9ee15aa0f2da9f9f0c48e13324e0e5f2e840)
|
| |
|
|
|
|
|
|
|
| |
While here make it return void, we don't set any useful return value
and nothing checks for it.
Sponsored by: Arm Ltd
(cherry picked from commit a7c5c88cb29ff512467a6599f1b4d8e2b8817480)
|
| |
|
|
|
|
|
|
|
|
|
| |
Report when SVE is present and allow it to be used by calling
sve_restore_state on an SVE exception from userspace.
Reviewed by: kib
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D43310
(cherry picked from commit 205c1007be6d57b2eb8620f79384cb2299dde40e)
|
| |
|
|
|
|
|
|
|
|
|
| |
As with floating point instructions don't trap SVE instructions to the
hypervisor. This lets us handle then in the kernel.
Reviewed by: imp (earlier version)
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D43303
(cherry picked from commit fe5ed2496e44aec018a6215175bba225b20d81fd)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
Add the NT_ARM_SVE note type and use it to access the SVE registers
from ptrace. This allows userspace to modify the full SVE register
values.
Try to follow the Linux semantics to allow debuggers to use this with
minimal changes.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D43309
(cherry picked from commit aab60068943d733b0b4573e5481c543ab3d45a00)
|
| |
|
|
|
|
|
|
|
|
|
| |
To allow for user space to read the SVE vector length add a sysarch
handler to return the value to userspace.
Reviewed by: imp
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D43308
(cherry picked from commit 87a58d931db7b40d8d225a7edb82b78937078b7f)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
Use the new extended register support in the arm64 signal frame to
handle the SVE registers.
As these registers alias the VFP registers we use the floating point
register values to restore the lower bits of the SVE registers. This
is to support software that doesn't understand SVE to continue working.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D43307
(cherry picked from commit e45132e978030f0aa964d975a9fbad5614f260bd)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Add initial kernel support for SVE. This detects if SVE is present on
all CPUs, and if so allows for the use of SVE in the future.
As the SVE registers are a superset of the VFP registers we don't need
to restore the VFP registers when SVE is enabled.
Ths interface to enable SVE is provided, but not used until SVE is
supported in signals and with ptrace.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D43306
(cherry picked from commit 332c426328dbb30a6b2e69d9b1e8298d77d85bd1)
|
| |
|
|
|
|
|
|
|
|
|
| |
This will be used when we add SVE support to reduce the registers
needed to be saved on context switch.
Reviewed by: imp
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D43305
(cherry picked from commit a30149b2a9c6ac5280523eea9570e5b5e5f1fdf8)
|
| |
|
|
|
|
|
|
|
|
|
| |
Rather than try to detect when vfp_save_state is called by savectx use
a separate function that sets up the pcb as needed.
Reviewed by: imp
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D43304
(cherry picked from commit 12257233e8fd94ab24e1a84ad87126af2a7be33b)
|
| |
|
|
|
|
|
|
| |
Reviewed by: emaste
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D46513
(cherry picked from commit 7a488d83b3af4d59946319b251a3a2060f18df40)
|
| |
|
|
|
|
|
|
|
|
|
| |
Monitor Debug Configuration Register provides EL2 configuration options
for self-hosted debug and the Performance Monitors Extension.
Reviewed by: andrew
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D46191
(cherry picked from commit 610348a90467980de0498fab8dfdddf221d7a604)
|
| |
|
|
|
|
|
|
|
|
|
| |
This will be used to support FEAT_LPA2 to allow more than 48 bits of
physical address space.
Reviewed by: alc, kib, markj
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D46392
(cherry picked from commit 7be11454edd4eca6cbd299cb5b486294f912ffbd)
|
| |
|
|
|
|
|
|
|
| |
This leads to an unallocated register. Fix the value to point to
ESR_EL1.
Sponsored by: Arm Ltd
(cherry picked from commit 8b017284319c46d349abee5573d4656bb377b0fd)
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
When adding VHE support we may need to access these registers using
the _EL12 or _EL1 name depending on if VHE is enabled or not.
Add the registers we will likely need to access from VHE so these can
be updated in the guest switcher code.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D45525
(cherry picked from commit 47361851bc2e292bd7c723ba5f57792a7794f1b4)
|
| |
|
|
|
|
|
|
|
|
|
| |
CONTEXTIDR_EL1 is used in debug and trace features to identify the
current process or context.
Reviewed by: andrew
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D45173
(cherry picked from commit 4f8ba1c9dd00e0ae20fa161e166715746b6d1c04)
|
| |
|
|
|
|
|
|
|
| |
We have the mechanism in place to support encoding system registers
explicitly, so use that rather than requiring LLVM 13+, which breaks our
current set of GitHub CI builds.
Fixes: 9eecef052155 ("Add an Armv8 rndr random number provider")
(cherry picked from commit 9560ac4b638edf688566f576adc65d3654f2240c)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The arm and arm64 implementations of dispatching IPIs via PIC_IPI_SEND
are almost identical, and entirely MI with the lone exception of a
single store barrier on arm64 (that is likely either redundant or needed
on arm too). Thus, de-duplicate this code by moving it to INTRNG as a
generic IPI glue framework. The ipi_* functions remain declared in MD
smp.h headers and implemented in MD code, but are trivial wrappers
around intr_ipi_send that could be made MI, at least for INTRNG ports,
at a later date.
Note that, whilst both arm and arm64 had an ii_send member in intr_ipi
to abstract over how to send interrupts,, they were always ultimately
using PIC_IPI_SEND, and so this complexity has been removed. A follow-up
commit will re-introduce the same flexibility by instead allowing a
device other than the root PIC to be registered as the IPI sender.
As part of this, strengthen a MAXCPU assertion that was missed in commit
2f0b059eeafc ("intrng: switch from MAXCPU to mp_ncpus") (which itself is
mis-titled).
Reviewed by: mmel, mhorne
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D35898
(cherry picked from commit fae8755f16ff5b9bdc32df046e0f16c0cbb48a29)
|
| |
|
|
|
|
|
|
|
|
|
|
| |
When targeting Armv8.1 we can assume FEAT_LSE is available and can use
the atomic instructions this provides without needing to check for
support first.
Reviewed by: imp, markj, emaste
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D46159
(cherry picked from commit 7a345763f96d86eee9ab578e64311bf452e58900)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Before entering the kernel exception level ensure sctlr_el2 and
sctlr_el1 are in a known state. The EOS flag needs to be set to ensure
an eret instruction is a context synchronization event.
Set spcr_el1 when entering the kernel from EL1 and use an eret
instruction to return to the caller. This ensures the CPU pstate is
consistent with the value in spcr_el1 as it is the only way to set it
directly.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D45528
(cherry picked from commit 034c83fd7d85f57193850a73cc0ac957a211f725)
|
| |
|
|
|
|
|
|
|
|
|
| |
When entering the kernel with the E2H field set the layout of the
cnthctl_el2 register changes. Use the correct field locations to enable
access to the counter and timer registers from EL1.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D45529
(cherry picked from commit 997511dffe651e1d2d708f37f2ced430a6ab3349)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
To support recent extensions to the Arm architecture we may need to
store more or larger registers when sending a signal.
To support this create a list of these extra registers. Userspace that
needs to access a register in the signal handler can then walk the list
to find the correct register struct and read/write its contents.
Reviewed by: kib, markj (earlier version)
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D43302
(cherry picked from commit 7e6437c08415ade75403f1ecad75167257e8ea3c)
|
| |
|
|
|
|
|
|
|
|
|
| |
Adjust the mair_el1 macro indentation to be consistent with the
surrounding macros.
Reviewed by: emaste
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D45524
(cherry picked from commit 86bafddd61aba115bc46bcf1d7e0afb125850b5f)
|
| |
|
|
|
|
|
|
|
|
|
| |
Add the pointer authentication registers to armreg.h. These will be
used to support pointer authentication in a kernel built with GCC.
Reviewed by: jhb
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D45262
(cherry picked from commit 57d714a23f5ce21e389d53636b8bc6c1b45d518e)
|
| |
|
|
|
|
|
|
|
|
|
|
| |
While clang can handle numbers with a UL suffix in assembly files
gcc/gas is unable to. Switch to use the UL macro for TCR_EL1 defines as
some are used in locore.S
Reviewed by: brooks, jhb
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D45261
(cherry picked from commit 29c1cf9860e531146220d9dc3596e4c79f91cfcd)
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
Bits [5:0] of PMBSR_MSS encodes either Buffer Status Code (BSC) or Fault
Status Code (FSC) depending on PMBSR_EC value.
Add PMBSR_MSS_{BSC,FSC} to cover this field.
Reviewed by: andrew
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D45172
(cherry picked from commit 10b3eac88db689d657c4d0d0716bcbdf240ff614)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
When the register is not defined in Armv8.0 i.e. added in a later
extension, like SPE added in v8.2, the alternative name format of:
S<op0>_<op1>_C<crn>_C<crm>_<op2>
should be used; otherwise, calls to {READ,WRITE}_SPECIALREG() will
fail.
Use the MRS_REG_ALT_NAME() macro for SPE changing hex to decimal as
required by the macro.
Reviewed by: andrew
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D45171
(cherry picked from commit f7bdaa103eb8906fc999c7fd5e8d6af440e26e6c)
|
| |
|
|
|
|
|
|
|
|
|
| |
These can be used even when the compiler is too old for the register
to be included.
Reviewed by: Zachary Leaf <zachary.leaf@arm.com>
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D45176
(cherry picked from commit d6d860c7ff5c4cbe9475d98000407d6f0ea84b47)
|
| |
|
|
|
|
|
|
| |
Reviewed by: jhibbits, kevans
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D45081
(cherry picked from commit e353ac0cfd5d155c01253ee17c2bf23f888cb7de)
|
| |
|
|
|
|
|
|
| |
Reviewed by: jhb
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D44353
(cherry picked from commit c802b486ddfd796ea35b770148af9a5b0cd0ee64)
|
| |
|
|
|
|
|
|
|
|
|
| |
Add a macro to find which bits from far_el2 are needed to be copied
to get the full intermediate physical address (IPA).
The hpfar_el2 register only contains a 4k aligned fault address. We
need to include the lower bits from far_el2 if we need the full
faulting IPA.
(cherry picked from commit b3bbec37ba039a46f7166f6fa3571f38da4253e9)
|
| |
|
|
| |
(cherry picked from commit 9c52f98c9f1abfe6577335522b6007659f759adc)
|
| |
|
|
|
|
|
| |
Add a macro to get the ESR_ELx ISS value when we trap accessing a
special register.
(cherry picked from commit 09ac9cf8971a0709bb8d5a3a703cd3dbff882b6f)
|
| |
|
|
|
|
| |
Sponsored by: Arm Ltd
(cherry picked from commit 64963dd2f0a2a47e94fb3f7c56198c475b9821a3)
|
| |
|
|
|
|
|
| |
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D43971
(cherry picked from commit 5e7941b6c979a55dc67baf316db6932c09b14022)
|
| |
|
|
|
|
|
|
|
|
|
| |
No functional change.
Reviewed by: markj
MFC after: 3 days
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D45322
(cherry picked from commit b5e17840de65ff196b2398211b68ce96d6919f2c)
|