aboutsummaryrefslogtreecommitdiff
path: root/sys/powerpc/booke
Commit message (Collapse)AuthorAgeFilesLines
* powerpc: Don't use cache to zero pagesJustin Hibbits2025-11-192-6/+4
| | | | | | | pmap_zero_page() may be called on uncached pages, so using the cache to zero uncached pages may trigger a fault. MFC after: 2 weeks
* powerpc: add missing headerDoug Moore2025-04-211-0/+1
| | | | | | Add missing vm/vm_radix.header. Fixes: b6cd84ca2d08 ("powerpc: replace tailq pointers with iterators")
* powerpc: replace tailq pointers with iteratorsDoug Moore2025-04-211-6/+9
| | | | | | | | Change architecture-specific code to use iterators rather than tailq pointers. Reviewed by: kib Differential Revision: https://reviews.freebsd.org/D49927
* libkern: add ilog2 macroDoug Moore2024-06-033-27/+0
| | | | | | | | | | | | | | | The kernel source contains several definitions of an ilog2 function; some are slower than necessary, and one of them is incorrect. Elimininate them all and define an ilog2 macro in libkern to replace them, in a way that is fast, correct for all argument types, and, in a GENERIC kernel, includes a check for an invalid zero parameter. Folks at Microsoft have verified that having a correct ilog2 definition for their MANA driver doesn't break it. Reviewed by: alc, markj, mhorne (older version), jhibbits (older version) Differential Revision: https://reviews.freebsd.org/D45170 Differential Revision: https://reviews.freebsd.org/D45235
* Adjust comments referencing vm_mem_init()Mitchell Horne2024-05-271-2/+3
| | | | | | | | | I cannot find a time where the function was not named this. Reviewed by: kib, markj MFC after: 3 days Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D45383
* powerpc/booke: Reserve KVA for minidump working areaJustin Hibbits2024-04-041-0/+5
| | | | | This was already handled for the AIM64 pmaps, so add the same to Book-E64 pmap.
* pmap: Convert boolean_t to bool.John Baldwin2024-01-313-29/+29
| | | | | Reviewed by: kib (older version) Differential Revision: https://reviews.freebsd.org/D39921
* sys: Automated cleanup of cdefs and other formattingWarner Losh2023-11-275-5/+1
| | | | | | | | | | | | | | | | Apply the following automated changes to try to eliminate no-longer-needed sys/cdefs.h includes as well as now-empty blank lines in a row. Remove /^#if.*\n#endif.*\n#include\s+<sys/cdefs.h>.*\n/ Remove /\n+#include\s+<sys/cdefs.h>.*\n+#if.*\n#endif.*\n+/ Remove /\n+#if.*\n#endif.*\n+/ Remove /^#if.*\n#endif.*\n/ Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/types.h>/ Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/param.h>/ Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/capsicum.h>/ Sponsored by: Netflix
* pmap: Prefer consistent naming for loader tunableZhenlei Huang2023-10-191-1/+1
| | | | | | | | | | | | | | The sysctl knob 'vm.pmap.pv_entry_max' becomes a loader tunable since 7ff48af7040f (Allow a specific setting for pv entries) but is fetched from system environment 'vm.pmap.pv_entries'. That is inconsistent and obscure. This reverts 36e1b9702e21 (Correct the tunable name in the message). PR: 231577 Reviewed by: jhibbits, alc, kib MFC after: 3 days Differential Revision: https://reviews.freebsd.org/D42274
* sys: Remove $FreeBSD$: one-line .c patternWarner Losh2023-08-168-16/+0
| | | | Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/
* sys: Remove $FreeBSD$: two-line .h patternWarner Losh2023-08-162-4/+0
| | | | Remove /^\s*\*\n \*\s+\$FreeBSD\$$\n/
* powerpc booke: Add an __unused wrapper for a variable only used under DEBUG.John Baldwin2023-05-251-1/+3
|
* spdx: The BSD-2-Clause-FreeBSD identifier is obsolete, drop -FreeBSDWarner Losh2023-05-126-6/+6
| | | | | | | | | The SPDX folks have obsoleted the BSD-2-Clause-FreeBSD identifier. Catch up to that fact and revert to their recommended match of BSD-2-Clause. Discussed with: pfg MFC After: 3 days Sponsored by: Netflix
* powerpc: Use valid prototypes for function declarations with no arguments.John Baldwin2023-04-241-3/+3
| | | | | Reviewed by: emaste Differential Revision: https://reviews.freebsd.org/D39733
* powerpc: fix a few pmap related functions to return correct typesDimitry Andric2023-04-231-1/+1
| | | | | | | | | | | While experimenting with changing boolean_t to another type, I noticed that several powerpc pmap related functions returned the wrong type: boolean_t instead of int. Fix several declarations and definitions to match the actual pmap function types: pmap_dev_direct_mapped_t and pmap_ts_referenced_t. MFC after: 3 days
* pmap_unmapdev/bios: Accept a pointer instead of a vm_offset_t.John Baldwin2022-09-221-3/+4
| | | | | | | | This matches the return type of pmap_mapdev/bios. Reviewed by: kib, markj Sponsored by: DARPA Differential Revision: https://reviews.freebsd.org/D36548
* powerpc: Correct some typos in source code commentsGordon Bergling2022-09-041-2/+2
| | | | | | - s/occured/occurred/ MFC after: 3 days
* Adjust function definitions in mp_cpudep.c.c to avoid clang 15 warningsDimitry Andric2022-08-151-2/+2
| | | | | | | | | | | | | | | | | | | With clang 15, the following -Werror warnings are produced: sys/powerpc/booke/mp_cpudep.c:54:20: error: a function declaration without a prototype is deprecated in all versions of C [-Werror,-Wstrict-prototypes] cpudep_ap_bootstrap() ^ void sys/powerpc/booke/mp_cpudep.c:97:16: error: a function declaration without a prototype is deprecated in all versions of C [-Werror,-Wstrict-prototypes] cpudep_ap_setup() ^ void This is because cpudep_ap_bootstrap() and cpudep_ap_setup() are declared with (void) argument lists, but defined with empty argument lists. Make the definitions match the declarations. MFC after: 3 days
* AST: reworkKonstantin Belousov2022-08-021-4/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | Make most AST handlers dynamically registered. This allows to have subsystem-specific handler source located in the subsystem files, instead of making subr_trap.c aware of it. For instance, signal delivery code on return to userspace is now moved to kern_sig.c. Also, it allows to have some handlers designated as the cleanup (kclear) type, which are called both at AST and on thread/process exit. For instance, ast(), exit1(), and NFS server no longer need to be aware about UFS softdep processing. The dynamic registration also allows third-party modules to register AST handlers if needed. There is one caveat with loadable modules: the code does not make any effort to ensure that the module is not unloaded before all threads processed through AST handler in it. In fact, this is already present behavior for hwpmc.ko and ufs.ko. I do not think it is worth the efforts and the runtime overhead to try to fix it. Reviewed by: markj Tested by: emaste (arm64), pho Discussed with: jhb Sponsored by: The FreeBSD Foundation MFC after: 1 week Differential revision: https://reviews.freebsd.org/D35888
* powerpc: Fix a typo in a commentGordon Bergling2022-06-041-1/+1
| | | | | | - s/layed/laid/ MFC after: 3 days
* powerpc booke cpu_machine_check: Remove unused variable.John Baldwin2022-04-181-3/+0
| | | | | Reviewed by: jhibbits Differential Revision: https://reviews.freebsd.org/D34902
* sched: separate out schedinit_ap()Kyle Evans2021-11-031-0/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | schedinit_ap() sets up an AP for a later call to sched_throw(NULL). Currently, ULE sets up some pcpu bits and fixes the idlethread lock with a call to sched_throw(NULL); this results in a window where curthread is setup in platforms' init_secondary(), but it has the wrong td_lock. Typical platform AP startup procedure looks something like: - Setup curthread - ... other stuff, including cpu_initclocks_ap() - Signal smp_started - sched_throw(NULL) to enter the scheduler cpu_initclocks_ap() may have callouts to process (e.g., nvme) and attempt to sched_add() for this AP, but this attempt fails because of the noted violated assumption leading to locking heartburn in sched_setpreempt(). Interrupts are still disabled until cpu_throw() so we're not really at risk of being preempted -- just let the scheduler in on it a little earlier as part of setting up curthread. Reviewed by: alfredo, kib, markj Triage help from: andrew, markj Smoke-tested by: alfredo (ppc), kevans (arm64, x86), mhorne (arm) Differential Revision: https://reviews.freebsd.org/D32797
* Convert vm_page_alloc() callers to use vm_page_alloc_noobj().Mark Johnston2021-10-202-8/+5
| | | | | | | | | | | | | | | Remove page zeroing code from consumers and stop specifying VM_ALLOC_NOOBJ. In a few places, also convert an allocation loop to simply use VM_ALLOC_WAITOK. Similarly, convert vm_page_alloc_domain() callers. Note that callers are now responsible for assigning the pindex. Reviewed by: alc, hselasky, kib MFC after: 1 week Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D31986
* Create sys/reg.h for the common code previously in machine/reg.hAndrew Turner2021-08-301-1/+0
| | | | | | | | | | Move the common kernel function signatures from machine/reg.h to a new sys/reg.h. This is in preperation for adding PT_GETREGSET to ptrace(2). Reviewed by: imp, markj Sponsored by: DARPA, AFRL (original work) Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D19830
* Clean up a couple of MD warts in vm_fault_populate():Jason A. Harmening2021-03-311-0/+8
| | | | | | | | | | | | | | | | | | | | --Eliminate a big ifdef that encompassed all currently-supported architectures except mips and powerpc32. This applied to the case in which we've allocated a superpage but the pager-populated range is insufficient for a superpage mapping. For platforms that don't support superpages the check should be inexpensive as we shouldn't get a superpage in the first place. Make the normal-page fallback logic identical for all platforms and provide a simple implementation of pmap_ps_enabled() for MIPS and Book-E/AIM32 powerpc. --Apply the logic for handling pmap_enter() failure if a superpage mapping can't be supported due to additional protection policy. Use KERN_PROTECTION_FAILURE instead of KERN_FAILURE for this case, and note Intel PKU on amd64 as the first example of such protection policy. Reviewed by: kib, markj, bdragon Differential Revision: https://reviews.freebsd.org/D29439
* [PowerPC] Fix 32-bit Book-E panic due to pve leakBrandon Bergren2021-03-291-3/+12
| | | | | | | | | | | | | | | | | | On an INVARIANTS kernel on 32-bit Book-E, we were panicing when running the libproc tests. This was caused by extra pv entries being generated accidentally by the pmap icache invalidation code. Use the same VA (i.e. 0) when freeing the temporary mapping, instead of some arbitrary address within the zero page. Failure to do this was causing kernel-side icache syncing to leak PVE entries when invalidating icache for a non page-aligned address, which would later result in pages erroneously showing up as mapped to vm_page. This bug was introduced in r347354 in 2019. Reviewed by: jhibbits (in irc) Sponsored by: Tag1 Consulting, Inc.
* Add CFI start/end proc directives to arm64, i386, and ppcConrad Meyer2020-12-052-0/+13
| | | | | | | | | | | | | | Follow-up to r353959 and r368070: do the same for other architectures. arm32 already seems to use its own .fnstart/.fnend directives, which appear to be ARM-specific variants of the same thing. Likewise, MIPS uses .frame directives. Reviewed by: arichardson Differential Revision: https://reviews.freebsd.org/D27387 Notes: svn path=/head/; revision=368354
* Avoid dump_avail[] redefinition.Konstantin Belousov2020-10-141-1/+2
| | | | | | | | | | | | | Move dump_avail[] extern declaration and inlines into a new header vm/vm_dumpset.h. This fixes default gcc build for mips. Reviewed by: alc, scottph Tested by: kevans (previous version) Sponsored by: The FreeBSD Foundation Differential revision: https://reviews.freebsd.org/D26741 Notes: svn path=/head/; revision=366711
* powerpc: clean up empty lines in .c and .h filesMateusz Guzik2020-09-016-35/+13
| | | | Notes: svn path=/head/; revision=365073
* Revert r240317 to prevent leaking pmap entriesConrad Meyer2020-07-161-0/+1
| | | | | | | | | | | | | | | | | | | | | | | Subsequent to r240317, kmem_free() was replaced with kva_free() (r254025). kva_free() releases the KVA allocation for the mapped region, but no longer clears the pmap (pagetable) entries. An affected pmap_unmapdev operation would leave the still-pmap'd VA space free for allocation by other KVA consumers. However, this bug easily avoided notice for ~7 years because most devices (1) never call pmap_unmapdev and (2) on amd64, mostly fit within the DMAP and do not need KVA allocations. Other affected arch are less popular: i386, MIPS, and PowerPC. Arm64, arm32, and riscv are not affected. Reported by: Don Morris <dgmorris AT earthlink.net> Submitted by: Don Morris (amd64 part) Reviewed by: kib, markj, Don (!amd64 parts) MFC after: I don't intend to, but you might want to Sponsored by: Dell Isilon Differential Revision: https://reviews.freebsd.org/D25689 Notes: svn path=/head/; revision=363266
* [PowerPC] Fix booke64 qemu infinite loop in L2 cache enableBrandon Bergren2020-06-191-1/+11
| | | | | | | | | | | | | | | | | | Since qemu does not implement the L2 cache, we get stuck forever waiting for a bit to be set when trying to invalidate it. To prevent that, we should bail out if the L2 cache is missing. One easy way to check this is L2CFG0 == 0 (since L2CSIZE always has at least one bit set in a valid implementation) (tested on qemu, rb800, and x5000) Reviewed by: jhibbits Sponsored by: Tag1 Consulting, Inc. Differential Revision: https://reviews.freebsd.org/D25225 Notes: svn path=/head/; revision=362407
* powerpc/pmap: Fix pte_find_next() iterators for booke64 pmapJustin Hibbits2020-06-101-2/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | After r361988 fixed the reference count leak on booke64, it became possible for an iteration somewhere in the middle of a page to become stale, with the page vanishing (correctly) due to all PTEs on that page going away. pte_find_next() would start at that iterator, and move along 'higher' order directory pages until it finds a valid one, without zeroing out the lower order pages. For instance: /* Find next pte at or above 0x10002000. */ pte = pte_find_next(pmap, &(0x10002000)); pte_remove(pmap, pte); /* This pte was the last reference in the page table page, page is * gone. */ pte = pte_find_next(pmap, 0x10002000); /* pte_find_next will see 0x10002000's page is gone, and jump to the * next one, but starting iteration at the '0x2000' slot, skipping * 0x0000 and 0x1000. */ This caused some processes, like git, to trip the KASSERT() in pmap_release(). Fix this by zeroing all lower order iterators at each level. Notes: svn path=/head/; revision=362034
* powerpc/pmap: Fix wired memory leak in booke64 page directoriesJustin Hibbits2020-06-091-18/+24
| | | | | | | | | | | | Properly handle reference counts in the 64-bit pmap page directories. Otherwise all page table pages would leak due to over-referencing. This would cause a quick enter to swap on a desktop system (AmigaOne X5000) when quitting and rerunning applications, or just building world. Add an INVARIANTS check to validate no leakage at pmap release time. Notes: svn path=/head/; revision=361988
* powerpc: Use IFUNCs for copyin/copyout/etcJustin Hibbits2020-06-061-23/+0
| | | | | | | | | | | | | | | | Summary: Radix on AIM, and all of Book-E (currently), can do direct addressing of user space, instead of needing to map user addresses into kernel space. Take advantage of this to optimize the copy(9) functions for this behavior, and avoid effectively NOP translations. Test Plan: Tested on powerpcspe, powerpc64/booke, powerpc64/AIM Reviewed by: bdragon Differential Revision: https://reviews.freebsd.org/D25129 Notes: svn path=/head/; revision=361861
* powerpc/mmu: Convert PowerPC pmap drivers to ifunc from kobjJustin Hibbits2020-05-274-304/+299
| | | | | | | | | | | | | With IFUNC support in the kernel, we can finally get rid of our poor-man's ifunc for pmap, utilizing kobj. Since moea64 uses a second tier kobj as well, for its own private methods, this adds a second pmap install function (pmap_mmu_init()) to perform pmap 'post-install pre-bootstrap' initialization, before the IFUNCs get initialized. Reviewed by: bdragon Notes: svn path=/head/; revision=361544
* powerpc/booke pmap: Fix iteration for 64-bit kernel page table creationJustin Hibbits2020-05-261-1/+2
| | | | | | | | | Kernel page tables actually start at index 4096, given kernel base address of 0xc008000000000000, not index 0, which would yield 0xc000000000000000. Fix this by indexing at the real base, instead of the assumed base. Notes: svn path=/head/; revision=361494
* powerpc64: Implement Radix MMU for POWER9 CPUsJustin Hibbits2020-05-111-0/+9
| | | | | | | | | | | | | | | | Summary: POWER9 supports two MMU formats: traditional hashed page tables, and Radix page tables, similar to what's presesnt on most other architectures. The PowerISA also specifies a process table -- a table of page table pointers-- which on the POWER9 is only available with the Radix MMU, so we can take advantage of it with the Radix MMU driver. Written by Matt Macy. Differential Revision: https://reviews.freebsd.org/D19516 Notes: svn path=/head/; revision=360887
* powerpc: Add a CPU-custom machine check handlerJustin Hibbits2020-05-101-0/+12
| | | | | | | | | | | | | | | | | Summary: Some machine checks are process-recoverable, others are not. Let a CPU-specific handler decide what to do. This works around a machine check error hit while building www/firefox and mail/thunderbird, which would otherwise cause the build to fail. More work is needed to handle all possible machine check conditions, but this is sufficient to unblock some ports building. Differential Revision: https://reviews.freebsd.org/D23731 Notes: svn path=/head/; revision=360881
* powerpc/booke: Use power-of-two mappings in 64-bit pmap_mapdevJustin Hibbits2020-04-111-4/+49
| | | | | | | | | | | | | | | | | | | | | Summary: This reduces the precious TLB1 entry consumption (64 possible in existing 64-bit cores), by adjusting the size and alignment of a device mapping to a power of 2, to encompass the full mapping and its surroundings. One caveat with this: If a mapping really is smaller than a power of 2, it's possible to get a machine check or hang if the 'missing' physical space is accessed. In practice this should not be an issue for users, as devices overwhelmingly have physical spaces on power-of-two sizes and alignments, and any design that includes devices which don't follow this can be addressed by undefining the POW2_MAPPINGS guard. Reviewed by: bdragon Differential Revision: https://reviews.freebsd.org/D24248 Notes: svn path=/head/; revision=359793
* powerpc/booke: Add pte_find_next() to find the next in-use PTEJustin Hibbits2020-04-113-3/+80
| | | | | | | | | | | | | | | | Summary: Iterating over VM_MIN_ADDRESS->VM_MAXUSER_ADDRESS can take a very long time iterating one page at a time (2**(log_2(SIZE)-12) operations), yielding possibly several days or even weeks on 64-bit Book-E, even for a largely empty, which can happen when swapping out a process by vmdaemon. Speed this up by instead finding the next PTE at or equal to the given VA. Reviewed by: bdragon Differential Revision: https://reviews.freebsd.org/D24238 Notes: svn path=/head/; revision=359792
* powerpc/booke: Change Book-E 64-bit pmap to 4-level tableJustin Hibbits2020-04-114-257/+235
| | | | | | | | | | | | | Summary: The existing page table is fraught with errors, since it creates a hole in the address space bits. Fix this by taking a cue from the POWER9 radix pmap, and make the page table 4 levels, 52 bits. Reviewed by: bdragon Differential Revision: https://reviews.freebsd.org/D24220 Notes: svn path=/head/; revision=359791
* powerpc/pmap: Replace a logical TAILQ_FOREACH_SAFE with the real thingJustin Hibbits2020-03-301-3/+1
| | | | | | | No functional change, just cleanup. Notes: svn path=/head/; revision=359445
* powerpc: Axe PPC4xx support.Justin Hibbits2020-03-182-222/+0
| | | | | | | | | | | | Summary: The support was added almost a decade ago, and never completed. Just axe it. It was also inadvertently broken 5 years ago, and nobody noticed. Reviewed by: bdragon Differential Revision: https://reviews.freebsd.org/D23753 Notes: svn path=/head/; revision=359058
* powerpc: Simplify _nodrop variants of FPU and vector register savesJustin Hibbits2020-03-131-4/+1
| | | | | | | | | No need for an extra temporary. It doesn't even help with readability. Suggested by: kib (almost 2 years ago) Notes: svn path=/head/; revision=358928
* powerpc/booke: Split out 32- and 64- bit pmap details from main bodyJustin Hibbits2020-03-103-1313/+1704
| | | | | | | | | | | | | | | | | | Summary: This is largely a straight-forward cleave of the 32-bit and 64-bit page table specifics, along with the mmu_booke_*() functions that are wholely different between the two implementations. The ultimate goal of this is to make it easier to reason about and update a specific implementation without wading through the other implementation details. This is in support of further changes to the 64-bit pmap. Reviewed by: bdragon Differential Revision: https://reviews.freebsd.org/D23983 Notes: svn path=/head/; revision=358828
* Unbreak the 32-bit powerpc buildsJustin Hibbits2020-02-251-1/+1
| | | | | | | | Force unsigned integer usage by casting to vm_offset_t, to avoid integer overflow, from r358305 Notes: svn path=/head/; revision=358306
* powerpc/booke: Use a pseudo-DMAP for the device mappings on booke64Justin Hibbits2020-02-251-13/+26
| | | | | | | | | | | | | | | Since powerpc64 has such a large virtual address space, significantly larger than its physical address space, take advantage of this, and create yet another DMAP-like instance for the device mappings. In this case, the device mapping "DMAP" is in the 0x8000000000000000 - 0xc000000000000000 range, so as not to overlap the physical memory DMAP. This will allow us to add TLB1 entry coalescing in the future, especially useful for things like the radeonkms driver, which maps parts of the GPU at a time, but eventually maps all of it, using up a lot of TLB1 entries (~40). Notes: svn path=/head/; revision=358305
* powerpc/booke: Fix handling of pvh_global_lock and pmap lockJustin Hibbits2020-02-221-7/+6
| | | | | | | | | | | | | | ptbl_alloc() is expected to return with the pvh_global_lock and pmap lock held. However, it will return with them unlocked if nosleep is specified. Along with this, fix lock ordering of pvh_global_lock with respect to the pmap lock in other places. Differential Revision: https://reviews.freebsd.org/D23692 Notes: svn path=/head/; revision=358234
* [PowerPC] Fix VSX context handlingBrandon Bergren2020-02-041-0/+18
| | | | | | | | | | | | | | | | | | | | | | In r356767, memcpy/memmove/bcopy optimizations were added to libc to improve performance. This exposed an existing kernel issue in VSX handling. The PSL_VSX flag was not being excluded from the psl_userstatic set, which meant that any thread that used these and then called swapcontext(3) would get an EINVAL error. Fixing this exposed a second issue - in r344123, the FPU was being forced off in set_mcontext(). However, this was neglecting to ensure VSX was turned off at the same time. While here, add some code comments to explain what's going on. Reviewed by: jhibbits, luporl (earlier rev), pkubaj (earlier rev) Sponsored by: Tag1 Consulting, Inc. Differential Revision: https://reviews.freebsd.org/D23497 Notes: svn path=/head/; revision=357529
* [PowerPC] Fix Book-E direct map for >=16G ram on e5500Brandon Bergren2020-01-181-1/+16
| | | | | | | | | | | | | | | | | It turns out the maximum TLB1 page size on e5500 is 4G, despite the format being defined for up to 1TB. So, we need to clamp the DMAP TLB1 entries to not attempt to create 16G or larger entries. Fixes boot on my X5000 in which I just installed 16G of RAM. Reviewed by: jhibbits Sponsored by: Tag1 Consulting, Inc. Differential Revision: https://reviews.freebsd.org/D23244 Notes: svn path=/head/; revision=356857
generated by cgit v1.2.3 (git 2.25.1) at 2026-01-01 13:18:20 +0000