aboutsummaryrefslogtreecommitdiff
path: root/sys/arm64/include/atomic.h
Commit message (Collapse)AuthorAgeFilesLines
* atomics: Constify loadsOlivier Certner2024-12-161-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | In order to match reality, allow using these functions with pointers on const objects, and bring us closer to C11. Remove the '+' modifier in the atomic_load_acq_64_i586()'s inline asm statement's constraint for '*p' (the value to load). CMPXCHG8B always writes back some value, even when the value exchange does not happen in which case what was read is written back. atomic_load_acq_64_i586() further takes care of the operation atomically writing back the same value that was read in any case. All in all, this makes the inline asm's write back undetectable by any other code, whether executing on other CPUs or code on the same CPU before and after the call to atomic_load_acq_64_i586(), except for the fact that CMPXCHG8B will trigger a #GP(0) if the memory address is part of a read-only mapping. This unfortunate property is however out of scope of the C abstract machine, and in particular independent of whether the 'uint64_t' pointed to is declared 'const' or not. Approved by: markj (mentor) MFC after: 5 days Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D46887
* atomic(9): Implement atomic_testand(clear|set)_ptrJohn Baldwin2024-11-191-0/+2
| | | | | | | | | | For current architectures, these are just aliases for the existing operation on the relevant scalar integer. Reviewed by: imp, kib Obtained from: CheriBSD Sponsored by: AFRL, DARPA Differential Revision: https://reviews.freebsd.org/D47631
* arm64: Expand the use of Armv8.1-A atomicsAndrew Turner2024-08-191-2/+3
| | | | | | | | | | When targeting Armv8.1 we can assume FEAT_LSE is available and can use the atomic instructions this provides without needing to check for support first. Reviewed by: imp, markj, emaste Sponsored by: Arm Ltd Differential Revision: https://reviews.freebsd.org/D46159
* sys: Remove $FreeBSD$: two-line .h patternWarner Losh2023-08-161-2/+0
| | | | Remove /^\s*\*\n \*\s+\$FreeBSD\$$\n/
* arm64 lib32: prepare arm64 headers to redirect to armMike Karels2023-07-251-0/+6
| | | | | | | | | | | | | | | | | | | | | | | In order to compile lib32 libraries and other 32-bit code on arm64, <machine/foo.h> needs to be redirected to an arm header rather than arm64 when building with -m32. Ifdef the arm64 headers that are installed in /usr/include/machine and used by user-level software (including references from /usr/include/*.h) so that if __arm__ is defined when including the arm64 version, <arm/foo.h> is included rather than using the rest of the file's contents. Some arm headers had no arm64 equivalent; headers were added just to do the redirection. These files use #error if __arm__ is not defined to guard against confusion. Also add an include/arm Makefile, and modify Makefiles as needed to install everything, including the arm files in /usr/include/arm. fenv.h comes from lib/msun/arm/fenv.h. The new arm64 headers are: acle-compat.h cpuinfo.h sysreg.h Reviewed by: jrtc27, imp Differential Revision: https://reviews.freebsd.org/D40944
* arm64: add KASAN supportKyle Evans2023-03-231-1/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | This entails: - Marking some obvious candidates for __nosanitizeaddress - Similar trap frame markings as amd64, for similar reasons - Shadow map implementation The shadow map implementation is roughly similar to what was done on amd64, with some exceptions. Attempting to use available space at preinit_map_va + PMAP_PREINIT_MAPPING_SIZE (up to the end of that range, as depicted in the physmap) results in odd failures, so we instead search the physmap for free regions that we can carve out, fragmenting the shadow map as necessary to try and fit as much as we need for the initial kernel map. pmap_bootstrap_san() is thus after pmap_bootstrap(), which still included some technically reserved areas of the memory map that needed to be included in the DMAP. The odd failure noted above may be a bug, but I haven't investigated it all that much. Initial work by mhorne with additional fixes from kevans and markj. Reviewed by: andrew, markj Sponsored by: Juniper Networks, Inc. Sponsored by: Klara, Inc. Differential Revision: https://reviews.freebsd.org/D36701
* Fix the sanitizer header checks on arm64Andrew Turner2022-02-231-1/+1
| | | | | | | We need to not include the MI _san.h files when builing some parts of the kernel. Fix these checks in the arm64 header files. Sponsored by: The FreeBSD Foundation
* Add the char and short atomic operations on arm64Andrew Turner2022-02-231-0/+58
| | | | Sponsored by: The FreeBSD Foundation
* Add atomic_testandset_acq_* on arm64Andrew Turner2021-12-211-14/+16
| | | | | | | | | | We only need to include sys/_atomic_subword.h on arm64 to provide atomic_testandset_acq_long. Add an implementation in the arm64 atomic.h based on the existing atomic_testandset macro. Reviewed by: kib Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D33587
* Rename _cscan_atomic.h and _cscan_bus.h to atomic_san.h and bus_san.hMark Johnston2021-03-081-1/+1
| | | | | | | | | | | | Other kernel sanitizers (KMSAN, KASAN) require interceptors as well, so put these in a more generic place as a step towards importing the other sanitizers. No functional change intended. MFC after: 1 week Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D29103
* arm64: fix mask in atomic_testand{set,clear}_64Ryan Libby2021-01-021-2/+2
| | | | | | | | | | | | | These macros generate both the 32- and 64-bit ops, but the mask was hard coded for 32-bit ops, causing the 64-bit ops always to affect only the low 32 bits. PR: 252324 Reported by: gbe, mmel Reviewed by: markj, mmel Tested by: mmel, rwatson Sponsored by: Dell EMC Isilon Differential Revision: https://reviews.freebsd.org/D27886
* Expand generic subword atomic primitivesConrad Meyer2020-03-251-0/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The goal of this change is to make the atomic_load_acq_{8,16}, atomic_testandset{,_acq}_long, and atomic_testandclear_long primitives available in MI-namespace. The second goal is to get this draft out of my local tree, as anything that requires a full tinderbox is a big burden out of tree. MD specifics can be refined individually afterwards. The generic implementations may not be ideal for your architecture; feel free to implement better versions. If no subword_atomic definitions are needed, the include can be removed from your arch's machine/atomic.h. Generic definitions are guarded by defined macros of the same name. To avoid picking up conflicting generic definitions, some macro defines are added to various MD machine/atomic.h to register an existing implementation. Include _atomic_subword.h in arm and arm64 machine/atomic.h. For some odd reason, KCSAN only generates some versions of primitives. Generate the _acq variants of atomic_load.*_8, atomic_load.*_16, and atomic_testandset.*_long. There are other questionably disabled primitives, but I didn't run into them, so I left them alone. KCSAN is only built for amd64 in tinderbox for now. Add atomic_subword implementations of atomic_load_acq_{8,16} implemented using masking and atomic_load_acq_32. Add generic atomic_subword implementations of atomic_testandset_long(), atomic_testandclear_long(), and atomic_testandset_acq_long(), using atomic_fcmpset_long() and atomic_fcmpset_acq_long(). On x86, add atomic_testandset_acq_long as an alias for atomic_testandset_long. Reviewed by: kevans, rlibby (previous versions both) Differential Revision: https://reviews.freebsd.org/D22963 Notes: svn path=/head/; revision=359311
* Dynamically select LSE-based atomic(9)s on arm64.Mark Johnston2020-02-031-7/+44
| | | | | | | | | | | | | | | | | | | | | | | | Once all CPUs are online, determine if they all support LSE atomics and set lse_supported to indicate this. For now the atomic(9) implementations are still always inlined, though it would be preferable to create out-of-line functions to avoid text bloat. This was not done here since big.little systems exist in which some CPUs implement LSE while others do not, and ifunc resolution must occur well before this scenario can be detected. It does seem unlikely that FreeBSD will ever run on such platforms, however, so converting atomic(9) to use ifuncs is probably a good next step. Add a LSE_ATOMICS arm64 kernel configuration option to unconditionally select LSE-based atomic(9) implementations when the target system is known. Reviewed by: andrew, kib MFC after: 1 month Sponsored by: The FreeBSD Foundation, Amazon (hardware) Differential Revision: https://reviews.freebsd.org/D23325 Notes: svn path=/head/; revision=357460
* Add LSE-based atomic(9) implementations.Mark Johnston2020-02-031-23/+138
| | | | | | | | | | | | | | | | | These make use of the cas*, ld* and swp instructions added in ARMv8.1. Testing shows them to be significantly more performant than LL/SC-based implementations. No functional change here since the wrappers still unconditionally select the _llsc variants. Reviewed by: andrew, kib MFC after: 1 month Submitted by: Ali Saidi <alisaidi@amazon.com> (original version) Differential Revision: https://reviews.freebsd.org/D23324 Notes: svn path=/head/; revision=357459
* Add wrappers for arm64 atomics.Mark Johnston2020-02-031-17/+72
| | | | | | | | | | | | | | | | Add a _llsc suffix for the existing LL/SC-based implementations and add trivial wrappers. This is in preparation for supporting LSE-based atomic(9) implementations. No functional change intended. Reviewed by: andrew, kib MFC after: 1 month Sponsored by: The FreeBSD Foundation, Amazon (hardware) Differential Revision: https://reviews.freebsd.org/D23323 Notes: svn path=/head/; revision=357458
* Provide a single implementation for each of the arm64 atomic(9) ops.Mark Johnston2020-02-031-492/+140
| | | | | | | | | | | | | | | Parameterize the macros by type width as well as acq/rel semantics. This makes modifying the implementations much less tedious and error-prone and makes it easier to support alternate LSE-based implementations. No functional change intended. Reviewed by: andrew, kib MFC after: 1 month Sponsored by: The FreeBSD Foundation, Amazon (hardware) Differential Revision: https://reviews.freebsd.org/D23322 Notes: svn path=/head/; revision=357457
* Add atomic_testandset/clear on arm64.Andrew Turner2020-01-091-0/+84
| | | | | | | | | | | These will reportedly be used in future uma changes. MFC after: 2 weeks Sponsored by: DARPA, AFRL Differential Revision: https://reviews.freebsd.org/D23019 Notes: svn path=/head/; revision=356550
* Add the 8 and 16 bit atomic load/store functions with a barrier on arm64.Andrew Turner2020-01-031-0/+50
| | | | | | | | | | Reviewed by: cem MFC after: 2 weeks Sponsored by: DARPA, AFRL Differential Revision: https://reviews.freebsd.org/D22966 Notes: svn path=/head/; revision=356316
* Port the NetBSD KCSAN runtime to FreeBSD.Andrew Turner2019-11-211-2/+8
| | | | | | | | | | | | | | | | Update the NetBSD Kernel Concurrency Sanitizer (KCSAN) runtime to work in the FreeBSD kernel. It is a useful tool for finding data races between threads executing on different CPUs. This can be enabled by enabling KCSAN in the kernel config, or by using the GENERIC-KCSAN amd64 kernel. It works on amd64 and arm64, however the later needs a compiler change to allow -fsanitize=thread that KCSAN uses. Sponsored by: DARPA, AFRL Differential Revision: https://reviews.freebsd.org/D22315 Notes: svn path=/head/; revision=354942
* Add more 8 and 16 bit variants of the the atomic(9) functions on arm64.Andrew Turner2019-11-071-2/+82
| | | | | | | | | | | These are direct copies of the 32 bit functions, adjusted ad needed. While here fix atomic_fcmpset_16 to use the valid load and store exclusive instructions. Sponsored by: DARPA, AFRL Notes: svn path=/head/; revision=354452
* arm64: fix atomic_fcmpset_16Emmanuel Vadot2018-05-281-1/+1
| | | | | | | | | newval needs to be uint16_t Reported by: andrew Notes: svn path=/head/; revision=334297
* arm64: Add atomic_fcmpset_8 and atomic_fcmpset_16Emmanuel Vadot2018-05-281-0/+48
| | | | | | | Reviewed by: cognet Notes: svn path=/head/; revision=334295
* Add atomic_load(9) and atomic_store(9) operations.Konstantin Belousov2017-12-191-0/+2
| | | | | | | | | | | | | | | | | | | | | | | They provide relaxed-ordered atomic access semantic. Due to the FreeBSD memory model, the operations are syntaxical wrappers around the volatile accesses. The volatile qualifier is used to ensure that the access not optimized out and in turn depends on the volatile semantic as implemented by supported compilers. The motivation for adding the operation is to help people coming from other systems or knowing the C11/C++ standards where atomics have special type and require use of the special access operations. It is still the case that FreeBSD requires plain load and stores of aligned integer types to be atomic. Suggested by: jhb Reviewed by: alc, jhb Sponsored by: The FreeBSD Foundation MFC after: 1 week Differential revision: https://reviews.freebsd.org/D13534 Notes: svn path=/head/; revision=326971
* Remove a blank line accidentally added in r320403.Andrew Turner2017-06-301-1/+0
| | | | Notes: svn path=/head/; revision=320497
* Some of the atomic_clear_* functions were incorrectly defined to be anAndrew Turner2017-06-271-3/+4
| | | | | | | | | | | atomic add. Correct these, fixing a NULL-pointer dereference in netgraph. PR: 220273 MFC after: 3 days Sponsored by: DARPA, AFRL Notes: svn path=/head/; revision=320403
* Implement atomic_fcmpset_* for arm and arm64.Olivier Houchard2017-01-281-0/+64
| | | | Notes: svn path=/head/; revision=312925
* Rework the atomic code to reduce the repetition. This merges some of theAndrew Turner2015-12-011-577/+195
| | | | | | | | | | | | | | | | atomic functions where they are almost identical, or have acquire/release semantics. While here clean these function up. The cbnz instruction doesn't change the condition flags so drop cc, however they should have memory added to the clobber list. Reviewed by: kib Sponsored by: ABT Systems Ltd Differential Revision: https://reviews.freebsd.org/D4318 Notes: svn path=/head/; revision=291580
* Add more atomic_swap_* functions.Andrew Turner2015-07-311-0/+19
| | | | | | | | Obtained from: ABT Systems Ltd Sponsored by: The FreeBSD Foundation Notes: svn path=/head/; revision=286128
* Add the atomic_thread_fence() family of functions with intent toKonstantin Belousov2015-07-081-0/+28
| | | | | | | | | | | | | | | | | | | | | | | | | | provide a semantic defined by the C11 fences with corresponding memory_order. atomic_thread_fence_acq() gives r | r, w, where r and w are read and write accesses, and | denotes the fence itself. atomic_thread_fence_rel() is r, w | w. atomic_thread_fence_acq_rel() is the combination of the acquire and release in single operation. Note that reads after the acq+rel fence could be made visible before writes preceeding the fence. atomic_thread_fence_seq_cst() orders all accesses before/after the fence, and the fence itself is globally ordered against other sequentially consistent atomic operations. Reviewed by: alc Discussed with: bde Sponsored by: The FreeBSD Foundation MFC after: 3 weeks Notes: svn path=/head/; revision=285283
* Add options to dmb() and dsb() macros on ARM64Zbigniew Bodek2015-06-091-6/+22
| | | | | | | | | | | | | | | Using plain dsb()/dmb() as full system barriers is usually to much. Adding proper options to those barriers (instead of full system - sy) will most likely reduce the cost of the instructions and will benefit in performance improvement. This commit adds options to barrier macro definitions. Obtained from: Semihalf Reviewed by: andrew, ian Sponsored by: The FreeBSD Foundation Notes: svn path=/head/; revision=284196
* Split out the _acq and _rel functions. These were the same, but there isAndrew Turner2015-04-061-43/+199
| | | | | | | | | | | | | | no need for them to be this strong, we only need to provide one or the other. While here replace atomic_load_acq_* and atomic_store_rel_* with a single instruction version, and fix the definition of atomic_clear_* to point to the correct functions. Sponsored by: The FreeBSD Foundation Notes: svn path=/head/; revision=281157
* Add the start of the arm64 machine headers. This is the subset needed toAndrew Turner2015-03-231-0/+574
start getting userland libraries building. Reviewed by: imp Sponsored by: The FreeBSD Foundation Notes: svn path=/head/; revision=280364
generated by cgit v1.3 (git 2.53.0) at 2026-10-27 03:55:08 +0000