path: root/sys/arm64/arm64/mp_machdep.c
Commit message (Collapse)AuthorAgeFilesLines
* Read the arm64 midr register earlierAndrew Turner2021-08-111-2/+1
| | | | | | | | We use the midr_el1 register to decode which CPU type we are booting from. Read it on the secondary CPUs before waiting for the boot CPU to release us as it will need to use it before the release. Sponsored by: The FreeBSD Foundation
* Remove "All Rights Reserved" from FreeBSD Foundation sys/ copyrightsEd Maste2021-08-081-1/+0
| | | | | | | These ones were unambiguous cases where the Foundation was the only listed copyright holder (in the associated license block). Sponsored by: The FreeBSD Foundation
* Enable IPIs on CPU 0 on arm and arm64Andrew Turner2021-05-021-0/+2
| | | | | | | | | | | Not all interrupt controllers enable IPIs by default as the Arm GIC specs make it an implementation defined option. As at least two hypervisors have also previously masked the IPIs on boot. As we already enable these IPIs on the non-boot CPUs it is expected this is a safe operation. Differential Revision: https://reviews.freebsd.org/D26975
* arm64: Include NUMA locality info in the CPU topologyMark Johnston2021-02-181-1/+28
| | | | | | | | | | | | | The scheduler uses this topology to try and preserve locality when migrating threads between CPUs and when performing work stealing. Ensure that on NUMA systems it will at least take the NUMA topology into account. Reviewed by: mmel Submitted by: Klara, Inc. Sponsored by: Ampere Computing MFC after: 1 week Differential Revision: https://reviews.freebsd.org/D28579
* Verify (and fix) the context_id argument passed to the mpentry () by PSCI.Michal Meloun2020-12-141-0/+15
| | | | | | | | | | | | Some older PSCI implementations corrupt (or do not pass) the context_id argument to newly started secondary cores. Although the ideal solution to this problem is u-boot update, we can find the correct value for the argument (cpuid) by comparing of real core mpidr register with the value stored in pcu->mpidr. MFC after: 2 weeks Notes: svn path=/head/; revision=368633
* Ensure the boot CPU is CPU 0 on arm64Andrew Turner2020-12-071-39/+66
| | | | | | | | | | | | | | | | | We assume the boot CPU is always CPU 0 on arm64. To allow for this reserve cpuid 0 for the boot CPU in the ACPI and FDT cases but otherwise start the CPU as normal. We then check for the boot CPU in start_cpu and return as if it was started. While here extract the FDT CPU init code into a new function to simplify cpu_mp_start and return FALSE from start_cpu when the CPU fails to start. Reviewed by: mmel Sponsored by: Innovate UK Differential Revision: https://reviews.freebsd.org/D27497 Notes: svn path=/head/; revision=368416
* Simplify startup of secondary cores and store MPIDR register to pcpu.Michal Meloun2020-12-051-67/+32
| | | | | | | | | | | | | | | | | | | | | | | | | - record MPIDR for all started cores in pcpu, they will be used as link between physical locality of given core, ID in external description (FDT or ACPI) and cupid. - because of above, cpuid can (and should) be freely assigned, only boot CPU must have cpuid 0. Simplify startup code according this. Please note that pure cpuid is not sufficient instrument to hold any information about core or cluster topology, nor to determistically iterate over subpart of cores in CPU (iterate over all cores in single cluster for example). Situation is more complicated by fact that PSCI can reject start of core without reporting error (because power budget for example), or by fact that is possible that we booted on non-first core in cluster (thus with cpuid 0 assigned to random core). Given cores topology should be exhibited to other parts of system (for example to scheduler for big.little or multicluster systems) by using smp_topo interface. Differential Revision: https://reviews.freebsd.org/D13863 Notes: svn path=/head/; revision=368370
* arm64: Make local stores observable before sending IPIsD Scott Phillips2020-08-261-0/+7
| | | | | | | | | | | | | | | | | | | | | | Add a synchronizing instruction to flush and wait until the local CPU's writes are observable to other CPUs before sending IPIs. This fixes an issue where recipient CPUs doing a rendezvous could enter the rendezvous handling code before the initiator's writes to the smp_rv_* variables were visible. This manifested as a system hang, where a single CPU's increment of smp_rv_waiters[0] actually happened "before" the initiator's zeroing of that field, so all CPUs were stuck with the field appearing to be at ncpus - 1. Reviewed by: andrew, markj Approved by: scottl (implicit) MFC after: 1 week Sponsored by: Ampere Computing, Inc. Differential Revision: https://reviews.freebsd.org/D25798 Notes: svn path=/head/; revision=364794
* Read the CPU 0 arm64 ID registers early in initarmAndrew Turner2020-07-011-1/+3
| | | | | | | | | | We also update the kernel view early in the boot. This will allow the use of the common kernel view in ifunc resolvers. Sponsored by: Innovate UK Notes: svn path=/head/; revision=362845
* Read the arm64 ID registers earlier in the boot process.Andrew Turner2020-07-011-6/+13
| | | | | | | | | | | | Also move parsing the registers to just after the secondary CPUs have started. This means the kernel register view from all CPUs is available after the CPU SYSINITs have finished, e.g. for use by ifunc resolvers. Sponsored by: Innovate UK Differential Revision: https://reviews.freebsd.org/D25505 Notes: svn path=/head/; revision=362837
* Call acpi_pxm_set_proximity_info() slightly earlier on x86.Mark Johnston2020-05-141-2/+0
| | | | | | | | | | | | | | | | | | This function is responsible for setting pc_domain in each pcpu structure. Call it from the main function that starts APs, rather than a separate SYSINIT. This makes it easier to close the window where UMA's per-CPU slab allocator may be called while pc_domain is uninitialized. In particular, the allocator uses pc_domain to allocate domain-local pages, so allocations before this point end up using domain 0 for everything. Reviewed by: kib MFC after: 1 week Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D24757 Notes: svn path=/head/; revision=361033
* Remove the secondary_stacks array in arm64 and riscv kernels.Mark Johnston2020-03-241-13/+55
| | | | | | | | | | | | | | | | | | | | | Instead, dynamically allocate a page for the boot stack of each AP when starting them up, like we do on x86. This shrinks the bss by MAXCPU*KSTACK_PAGES pages, which corresponds to 4MB on arm64 and 256KB on riscv. Duplicate the logic used on x86 to free the bootstacks, by using a sysinit to wait for each AP to switch to a thread before freeing its stack. While here, mark some static MD variables as such. Reviewed by: kib MFC after: 1 month Sponsored by: Juniper Networks, Klara Inc. Differential Revision: https://reviews.freebsd.org/D24158 Notes: svn path=/head/; revision=359280
* arm64: Don't enable interrupts in init_secondary().Mark Johnston2020-01-231-6/+0
| | | | | | | | | | | | | | | | Doing so can cause deadlocks or panics during boot, if an interrupt handler accesses uninitialized per-CPU scheduler structures. This seems to occur frequently when running under QEMU or AWS. The idle threads are set up to release a spinlock section and enable interrupts in fork_exit(), so there is no need to enable interrupts earlier. Reviewed by: kib MFC after: 1 week Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D23328 Notes: svn path=/head/; revision=357048
* Port the NetBSD KCSAN runtime to FreeBSD.Andrew Turner2019-11-211-0/+3
| | | | | | | | | | | | | | | | Update the NetBSD Kernel Concurrency Sanitizer (KCSAN) runtime to work in the FreeBSD kernel. It is a useful tool for finding data races between threads executing on different CPUs. This can be enabled by enabling KCSAN in the kernel config, or by using the GENERIC-KCSAN amd64 kernel. It works on amd64 and arm64, however the later needs a compiler change to allow -fsanitize=thread that KCSAN uses. Sponsored by: DARPA, AFRL Differential Revision: https://reviews.freebsd.org/D22315 Notes: svn path=/head/; revision=354942
* Utilize ASIDs to reduce both the direct and indirect costs of contextAlan Cox2019-11-031-0/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | switching. The indirect costs being unnecessary TLB misses that are incurred when ASIDs are not used. In fact, currently, when we perform a context switch on one processor, we issue a broadcast TLB invalidation that flushes the TLB contents on every processor. Mark all user-space ("ttbr0") page table entries with the non-global flag so that they are cached in the TLB under their ASID. Correct an error in pmap_pinit0(). The pointer to the root of the page table was being initialized to the root of the kernel-space page table rather than a user-space page table. However, the root of the page table that was being cached in process 0's md_l0addr field correctly pointed to a user-space page table. As long as ASIDs weren't being used, this was harmless, except that it led to some unnecessary page table switches in pmap_switch(). Specifically, other kernel processes besides process 0 would have their md_l0addr field set to the root of the kernel-space page table, and so pmap_switch() would actually change page tables when switching between process 0 and other kernel processes. Implement a workaround for Cavium erratum 27456 affecting ThunderX machines. (I would like to thank andrew@ for providing the code to detect the affected machines.) Address integer overflow in the definition of TCR_ASID_16. Setup TCR according to the PARange and ASIDBits fields from ID_AA64MMFR0_EL1. Previously, TCR_ASID_16 was unconditionally set. Modify build_l1_block_pagetable so that lower attributes, such as ATTR_nG, can be specified as a parameter. Eliminate some unused code. Earlier versions were tested to varying degrees by: andrew, emaste, markj MFC after: 3 weeks Differential Revision: https://reviews.freebsd.org/D21922 Notes: svn path=/head/; revision=354286
* Update the debug monitor handling to work after userspace has startedAndrew Turner2019-10-301-0/+6
| | | | | | | | | | | | The debug monitor register state is now stored in a struct and updated when required. Currently there is only a kernel state, however a per-process state will be added in a future change. Sponsored by: DARPA, AFRL Differential Revision: https://reviews.freebsd.org/D22128 Notes: svn path=/head/; revision=354175
* Centralize __pcpu definitions.Konstantin Belousov2019-08-291-2/+1
| | | | | | | | | | | | | | | | | | | | | | | | | Many extern struct pcpu <something>__pcpu declarations were copied/pasted in sources. The issue is that the definition is MD, but it cannot be provided by machine/pcpu.h due to actual struct pcpu defined in sys/pcpu.h later than the inclusion of machine/pcpu.h. This forced the copying when other code needed direct access to __pcpu. There is no way around it, due to machine/pcpu.h supplying part of struct pcpu fields. To work around the problem, add a new machine/pcpu_aux.h header, which should fill any needed MD definitions after struct pcpu definition is completed. This allows to remove copies of __pcpu spread around the source. Also on x86 it makes it possible to remove work arounds like OFFSETOF_CURTHREAD or clang specific warnings supressions. Reported and tested by: lwhsu, bcran Reviewed by: imp, markj (previous version) Discussed with: jhb Sponsored by: The FreeBSD Foundation Differential revision: https://reviews.freebsd.org/D21418 Notes: svn path=/head/; revision=351594
* Include ktr.h in more compilation unitsConrad Meyer2019-05-211-0/+1
| | | | | | | | | | | | | | | | | | Similar to r348026, exhaustive search for uses of CTRn() and cross reference ktr.h includes. Where it was obvious that an OS compat header of some kind included ktr.h indirectly, .c files were left alone. Some of these files clearly got ktr.h via header pollution in some scenarios, or tinderbox would not be passing prior to this revision, but go ahead and explicitly include it in files using it anyway. Like r348026, these CUs did not show up in tinderbox as missing the include. Reported by: peterj (arm64/mp_machdep.c) X-MFC-With: r347984 Sponsored by: Dell EMC Isilon Notes: svn path=/head/; revision=348064
* Add the hw.ncpu tunable to arm64.Andrew Turner2019-02-281-8/+11
| | | | | | | | | | | | This allows us to limit the number of CPUs to use, e.g. to debug problems seen when enabling multiple clusters. Reviewed by: manu Sponsored by: DARPA, AFRL Differential Revision: https://reviews.freebsd.org/D19404 Notes: svn path=/head/; revision=344659
* arm64: add ACPI based NUMA supportJayachandran C.2018-12-081-2/+10
| | | | | | | | | | | Use the newly defined SRAT/SLIT parsing APIs in arm64 to support ACPI based NUMA. Reviewed by: markj Differential Revision: https://reviews.freebsd.org/D17943 Notes: svn path=/head/; revision=341744
* Always set the MP_QUIRK_CPULIST quirk under ACPI. This needs a run timeAndrew Turner2018-10-311-0/+1
| | | | | | | | | | check to only set it for emulators as the CPU list may be changed when the emulator starts. Until this is working just always set it. Sponsored by: DARPA, AFRL Notes: svn path=/head/; revision=339961
* Remove function prototypes for functions removed in r339943.Andrew Turner2018-10-311-4/+0
| | | | | | | Sponsored by: DARPA, AFRL Notes: svn path=/head/; revision=339945
* Remove the unused arm64_cpu driver.Andrew Turner2018-10-311-74/+0
| | | | | | | | | | This was previously used for CPU initilisation, however this hasn't been the case in a long time. Sponsored by: DARPA, AFRL Notes: svn path=/head/; revision=339943
* Make it possible to disable NUMA support with a tunable.Mark Johnston2018-10-221-5/+6
| | | | | | | | | | | | | | | | | | | | This provides a chicken switch for anyone negatively impacted by enabling NUMA in the amd64 GENERIC kernel configuration. With NUMA disabled at boot-time, information about the NUMA topology is not exposed to the rest of the kernel, and all of physical memory is viewed as coming from a single domain. This method still has some performance overhead relative to disabling NUMA support at compile time. PR: 231460 Reviewed by: alc, gallatin, kib MFC after: 1 week Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D17439 Notes: svn path=/head/; revision=339616
* Eliminate the arena parameter to kmem_free(). Implicitly this corrects anAlan Cox2018-08-251-2/+1
| | | | | | | | | | | | | | | | | | | | | | error in the function hypercall_memfree(), where the wrong arena was being passed to kmem_free(). Introduce a per-page flag, VPO_KMEM_EXEC, to mark physical pages that are mapped in kmem with execute permissions. Use this flag to determine which arena the kmem virtual addresses are returned to. Eliminate UMA_SLAB_KRWX. The introduction of VPO_KMEM_EXEC makes it redundant. Update the nearby comment for UMA_SLAB_KERNEL. Reviewed by: kib, markj Discussed with: jeff Approved by: re (marius) Differential Revision: https://reviews.freebsd.org/D16845 Notes: svn path=/head/; revision=338318
* Eliminate kmem_malloc()'s unused arena parameter. (The arena parameterAlan Cox2018-08-211-2/+1
| | | | | | | | | | | | became unused in FreeBSD 12.x as a side-effect of the NUMA-related changes.) Reviewed by: kib, markj Discussed with: jeff, re@ Differential Revision: https://reviews.freebsd.org/D16825 Notes: svn path=/head/; revision=338143
* Allow releasing APs to take more time, as long as we are making progress.Andrew Turner2018-02-281-3/+14
| | | | | | | | | | | On large core count machines this can be slow while all the CPUs update the online counter. Sponsored by: DARPA, AFRL Sponsored by: Cavium (Hardware) Notes: svn path=/head/; revision=330120
* Add a framework to install CPU errata on arm64. Each erratum can encodeAndrew Turner2018-01-091-0/+1
| | | | | | | | | | | | | | a mask and value to compare with the Main ID Register. If these match then a function is called to handle the installation of the erratum workaround. No errata are currently handled, however this will change soon in a future commit. MFC after: 1 week Sponsored by: DARPA, AFRL Notes: svn path=/head/; revision=327727
* Add a memory barrier to ensure the atomic write is visible to the otherAndrew Turner2017-10-021-1/+4
| | | | | | | | | CPUs before waking them up. Sponsored by: DARPA, AFRL Notes: svn path=/head/; revision=324207
* Add support for quirks while enabling secondary CPUs. This uses the fdtAndrew Turner2017-08-311-1/+33
| | | | | | | | | | | | | | | | | | compatible string to check if the board is compatible with a given quirk. It's possible this will be moved later, however as it's currently only used by the MP code put it there. So far the only instance of a quirk is when the list of CPUs may be incorrect. This can happen on virtual machines with a hard coded devicetree, but where the user may then set the number of CPUs as an argument. This is the case on the ARM models so include the model specific compat strings for these, including the spelling mistake found in some of the OpenplatformPkg dtb files. Sponsored by: DARPA, AFRL Notes: svn path=/head/; revision=323070
* Reliably enable debug exceptions on all CPUs.John Baldwin2017-08-121-2/+1
| | | | | | | | | | | | | | | | | | | | | | Previously, debug exceptions were only enabled on the boot CPU if DDB was enabled in the dbg_monitor_init() function. APs also called this function, but since mp_machdep.c doesn't include opt_ddb.h, the APs ended up calling an empty stub defined in <machine/debug_monitor.h> instead of the real function. Also, if DDB was not enabled in the kernel, the boot CPU would not enable debug exceptions. Fix this by adding a new dbg_init() function that always clears the OS lock to enable debug exceptions which the boot CPU and the APs call. This function also calls dbg_monitor_init() to enable hardware breakpoints from DDB on all CPUs if DDB is enabled. Eventually base support for hardware breakpoints/watchpoints will need to move out of the DDB-only debug_monitor.c for use by userland debuggers. Reviewed by: andrew Differential Revision: https://reviews.freebsd.org/D12001 Notes: svn path=/head/; revision=322437
* Mark each cpu in the appropriate cpuset_domain set. This allows devices toAndrew Turner2017-08-051-5/+12
| | | | | | | | | | | | | handle cases where they can only run on a single domain. To allow all devices access to this set we need to move reading the domain earlier in the boot as it was previously handled in the CPU driver, however this is too late for the GICv3 ITS driver. Sponsored by: DARPA, AFRL Notes: svn path=/head/; revision=322109
* Read the numa-node-id property from each CPU node. This will initially beAndrew Turner2017-08-041-0/+5
| | | | | | | | | | used to support the dual package ThunderX where we need to send MSI/MSI-X interrupts to the same package as the device the interrupt came from. Sponsored by: DARPA, AFRL Notes: svn path=/head/; revision=322051
* In ARMv8.1 ARM has added a process state bit to disable access to userspaceAndrew Turner2017-04-131-0/+1
| | | | | | | | | | | | from the kernel. Make use of this to restrict accessing userspace to just the functions that explicitly handle crossing the user kernel boundary. Reported by: kib Sponsored by: DARPA, AFRL Differential Revision: https://reviews.freebsd.org/D10371 Notes: svn path=/head/; revision=316756
* Remove a static function declaration for a function not implemented.Bjoern A. Zeeb2017-01-231-2/+0
| | | | | | | | | Makes head code compile on 10.3 and cleanup is never wrong. MFC after: 3 days Notes: svn path=/head/; revision=312668
* Add ACPI support to the arm64 mp code. We use the Multiple APIC DescriptionAndrew Turner2016-12-071-18/+150
| | | | | | | | | | | Table to find the CPUs to find the CPUs to start. Currently we assume PSCI, however this assumption is shared with the FDT code. Obtained from: ABT Systems Ltd Sponsored by: The FreeBSD Foundation Notes: svn path=/head/; revision=309675
* We only use the cpu0 variable in the FDT code.Andrew Turner2016-11-291-0/+2
| | | | | | | | Obtained from: ABT Systems Ltd Sponsored by: The FreeBSD Foundation Notes: svn path=/head/; revision=309296
* Mark cpu_find_cpu0_fdt as FDT only. It's only called when this is set, andAndrew Turner2016-11-231-0/+2
| | | | | | | | | | the kernel is using FDT. Obtained from: ABT Systems Ltd Sponsored by: The FreeBSD Foundation Notes: svn path=/head/; revision=309048
* Remove the unneeded ofw_cpu_reg function signature, it's not used in thisAndrew Turner2016-11-231-2/+0
| | | | | | | | | | file. Obtained from: ABT Systems Ltd Sponsored by: The FreeBSD Foundation Notes: svn path=/head/; revision=309046
* Only release CPUs when they exist.Andrew Turner2016-10-261-0/+4
| | | | | | | | MFC after: 1 week Sponsored by: DARPA, AFRL Notes: svn path=/head/; revision=307961
* Create a new PSCI error code and use it to signal that starting the CPU isAndrew Turner2016-10-251-3/+7
| | | | | | | | | impossible as the PSCI firmware is missing. Sponsored by: ABT Systmes Ltd Notes: svn path=/head/; revision=307910
* Move printing the AArch64 ID registers to a new SYSINIT, the previousAndrew Turner2016-10-131-8/+2
| | | | | | | | | | location only prints them when booting on SMP with multiple cores. MFC after: 1 week Sponsored by: DARPA, AFRL Notes: svn path=/head/; revision=307210
* Add a kernel variable to let the user to select their preferred orderAndrew Turner2016-08-011-19/+15
| | | | | | | | | | | | | | | | | | | | | | between ACPI and FDT. This will be needed on machines with both, e.g. the SoftIron Overdrive 3000. The kernel will accept one or more comma separated values of either 'acpi' or 'fdt'. Any other values are skipped. To set it the user can either set it on the loader command line, or in loader.conf e.g. in loader.conf: kern.cfg.order=acpi,fdt This will try using ACPI then FDT. If none of the selected options work the kernel tries to use one to get the serial console, then panics. Reviewed by: emaste (earlier version) Obtained from: ABT Systems Ltd MFC after: 1 month Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D7274 Notes: svn path=/head/; revision=303614
* Remove an unused variable.Andrew Turner2016-07-251-10/+0
| | | | | | | | | Obtained from: ABT Systems Ltd MFC after: 1 month Sponsored by: The FreeBSD Foundation Notes: svn path=/head/; revision=303309
* Fix a typo in a string in a KASSERT to sanity check the CPU IDs.Andrew Turner2016-07-251-1/+1
| | | | | | | | | Obtained from: ABT Systems Ltd MFC after: 1 month Sponsored by: The FreeBSD Foundation Notes: svn path=/head/; revision=303308
* Rework how we number CPUs on arm64 to try and keep clusters together.Andrew Turner2016-07-251-1/+8
| | | | | | | | | Obtained from: ABT Systems Ltd MFC after: 1 month Sponsored by: The FreeBSD Foundation Notes: svn path=/head/; revision=303307
* Finish removing the non-INTRNG support from sys/arm64.Andrew Turner2016-07-141-61/+0
| | | | | | | | | Obtained from: ABT Systems Ltd MFC after: 1 month Sponsored by: The FreeBSD Foundation Notes: svn path=/head/; revision=302853
* dpcpu_init should have also passed in the calculated cpuid, not theAndrew Turner2016-05-311-1/+1
| | | | | | | | | | devicetree ID. Obtained from: ABT Systems Ltd Sponsored by: The FreeBSD Foundation Notes: svn path=/head/; revision=301072
* Allow the kernel to boot on a CPU where the devicetree has numbered it withAndrew Turner2016-05-311-12/+47
| | | | | | | | | | | a non-zero ID. To do this we increment the cpuid of any CPUs with a smaller devicetree ID by one to stop them conflicting with the boot CPU. Obtained from: ABT Systems Ltd Sponsored by: The FreeBSD Foundation Notes: svn path=/head/; revision=301070
* Move the call to intr_pic_init_secondary to the same place as in theAndrew Turner2016-05-161-5/+3
| | | | | | | | | | non-intrng case. Obtained from: ABT Systems Ltd Sponsored by: The FreeBSD Foundation Notes: svn path=/head/; revision=299939