aboutsummaryrefslogtreecommitdiff
path: root/sys/libkern
Commit message (Collapse)AuthorAgeFilesLines
* arm64: Mark the armv8 crc32c as supporting BTIAndrew Turner2024-02-191-0/+4
| | | | | | | | | | | | | This is built into a test so needs the BTI elf note for the rest of the test to be built with BTI support enabled. As the assembly uses the ENTRY macro it already supports BTI, so mark it as such. Sponsored by: Arm Ltd Differential Revision: https://reviews.freebsd.org/D42595 (cherry picked from commit 25d0310ed881e86cec9ab5cf37dd8471e88ad554)
* sys: Remove $FreeBSD$: one-line sh patternWarner Losh2023-08-231-1/+0
| | | | | | | Remove /^\s*#[#!]?\s*\$FreeBSD\$.*$\n/ Similar commit in current: (cherry picked from commit 031beb4e239b)
* sys: Remove $FreeBSD$: one-line .c patternWarner Losh2023-08-2379-157/+0
| | | | | | | Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/ Similar commit in current: (cherry picked from commit 685dc743dc3b)
* sys: Remove $FreeBSD$: one-line .h patternWarner Losh2023-08-232-2/+0
| | | | | | | Remove /^\s*\*+\s*\$FreeBSD\$.*$\n/ Similar commit in current: (cherry picked from commit 2ff63af9b88c)
* sys: Remove $FreeBSD$: two-line .h patternWarner Losh2023-08-234-8/+0
| | | | | | | Remove /^\s*\*\n \*\s+\$FreeBSD\$$\n/ Similar commit in current: (cherry picked from commit 95ee2897e98f)
* spdx: The BSD-2-Clause-FreeBSD identifier is obsolete, drop -FreeBSDWarner Losh2023-07-2516-16/+16
| | | | | | | | | | | The SPDX folks have obsoleted the BSD-2-Clause-FreeBSD identifier. Catch up to that fact and revert to their recommended match of BSD-2-Clause. Discussed with: pfg MFC After: 3 days Sponsored by: Netflix (cherry picked from commit 4d846d260e2b9a3d4d0a701462568268cbfe7a5b)
* Remove obsolete code gated on _ARM_ARCH_*Andrew Turner2023-01-241-29/+0
| | | | | | | | | This is all code only run on ARMv4 and ARMv5. Support for these have been dropped from FreeBSD. Differential Revision: https://reviews.freebsd.org/D28314 (cherry picked from commit 4e76e4c301c72698e111580e278872a8323fba50)
* qdivrem: Predict division by zero as false.Hans Petter Selasky2022-10-121-1/+1
| | | | | | | | | | | Division by zero triggers an arithmetic exception and should not be very common. Predict this. No functional change intended. Sponsored by: NVIDIA Networking (cherry picked from commit 1024bb26337bdc6679af477977247e9155d502bc)
* libkern: Fix a typo in a source code commentGordon Bergling2022-06-101-1/+1
| | | | | | | | - s/involes/involves/ Obtained from: NetBSD (cherry picked from commit 5cacfa7092f851ce31d75cd5dde73d26ea734497)
* libkern: Fix two typos in source code commentsGordon Bergling2022-04-141-2/+2
| | | | | | - s/noticably/noticeably/ (cherry picked from commit d10abf84ad9f508e5e58ec76f90e960eef228b3f)
* Remove "All Rights Reserved" from FreeBSD Foundation sys/ copyrightsEd Maste2022-02-081-1/+0
| | | | | | | | | These ones were unambiguous cases where the Foundation was the only listed copyright holder (in the associated license block). Sponsored by: The FreeBSD Foundation (cherry picked from commit 9feff969a01044c3083b552f06f7eb6416bc0524)
* Switch to an ifunc in the kernel for crc32cAndrew Turner2021-08-301-18/+38
| | | | | | | | | | | There is no need to read the same variable to check if the CPU supports crc32c instructions. Reviewed by: arichardson, kib, markj Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D31274 (cherry picked from commit a93941b439fce7047dffad6bc380cc9454b967cd)
* Remove incorrect __restricted labels from strcspnAlfonso Gregory2021-07-211-1/+1
| | | | | | | | | | | strcspn should never have had the __restrict keywords. While both of these strings are const, it may have unindended side effects. While this is the kernel, the POSIX definition also omits restrict. Reviewed by: imp@ Pull Request: https://github.com/freebsd/freebsd-src/pull/497 (cherry picked from commit 56d33e86b74b197a36f42255824b56715c96a596)
* Add missing libkern/strnstr.c fileGreg V2021-07-181-0/+64
| | | | | | | | | Apparently it wasn't picked up by git. Reviewed by: bz, hselasky, nc Differential Revision: https://reviews.freebsd.org/D31131 (cherry picked from commit 4211457e40e07f6f820c4171c7db81f028fd23af)
* Use '.arch_extension crc' in the arm64 crc32 codeAndrew Turner2021-06-121-1/+2
| | | | | | | | | We don't care about the base architecture here, just that the crc extension is enabled. Sponsored by: Innovate UK (cherry picked from commit 0ec3e991112d85a790ca3bbb4175652f37f4bd15)
* tests/sys/kern/crc32: Check for SSE4.2 before using itAlex Richardson2021-04-221-5/+11
| | | | | | | | | | | | | This avoids a SIGILL when running these tests on QEMU (which defaults to a basic amd64 CPU without SSE4.2). This commit also tests the table-based implementations in addition to testing the hw-accelerated crc32 versions. Reviewed By: cem, kib, markj Differential Revision: https://reviews.freebsd.org/D28395 (cherry picked from commit 83c20b8a2da04937cf4af127366b3dc92c855784)
* random(9): Restore historical [0,2^31-1] output range and related man ↵Lawrence Stewart2021-03-251-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | documention. Commit SVN r364219 / Git 8a0edc914ffd changed random(9) to be a shim around prng32(9) and inadvertently caused random(9) to begin returning numbers in the range [0,2^32-1] instead of [0,2^31-1], where the latter has been the documented range for decades. The increased output range has been identified as the source of numerous bugs in code written against the historical output range e.g. ipfw "prob" rules and stats(3) are known to be affected, and a non-exhaustive audit of the tree identified other random(9) consumers which are also likely affected. As random(9) is deprecated and slated for eventual removal in 14.0, consumers should gradually be audited and migrated to prng(9). Submitted by: Loic Prylli <lprylli@netflix.com> Obtained from: Netflix Reviewed by: cem, delphij, imp MFC after: 1 day MFC to: stable/13, releng/13.0 Differential Revision: https://reviews.freebsd.org/D29385 (cherry picked from commit dbbf3e3f37d67d3eae0931855f8b62b9b299b80a)
* Revert "Reimplement strlen"Mateusz Guzik2021-02-031-25/+54
| | | | | | | | | | | This reverts commit 710e45c4b8539d028877769f1a4ec088c48fb5f1. It breaks for some corner cases on big endian ppc64. Given the stage of the release process it is best to revert for now. Reported by: jhibbits (cherry picked from commit 33f0540b13d949c7cc226a79927ddc2062ff98bf)
* Reimplement strlenMateusz Guzik2021-02-011-54/+25
| | | | | | | | | | | | | | | | | | | | | | | | | The previous code neglected to use primitives which can find the end of the string without having to branch on every character. While here augment the somewhat misleading commentary -- strlen as implemented here leaves performance on the table, especially so for userspace. Every arch should get a dedicated variant instead. In the meantime this commit lessens the problem. Tested with glibc test suite. Naive test just calling strlen in a loop on Haswell (ops/s): $(perl -e "print 'A' x 3"): before: 211198039 after: 338626619 $(perl -e "print 'A' x 100"): before: 83151997 after: 98285919 (cherry picked from commit 710e45c4b8539d028877769f1a4ec088c48fb5f1)
* libkern/strcasestr.c: Drop xlocale support and connect to build.Vladimir Kondratyev2021-01-071-14/+8
| | | | | Reviewed by: markj, hselasky Differential revision: https://reviews.freebsd.org/D27866
* Copy strcasestr.c from libc to libkern.Vladimir Kondratyev2021-01-071-0/+74
|
* arc4random(9): Integrate with RANDOM_FENESTRASX push-reseedConrad Meyer2020-10-101-2/+36
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | There is no functional change for the existing Fortuna random(4) implementation, which remains the default in GENERIC. In the FenestrasX model, when the root CSPRNG is reseeded from pools due to an (infrequent) timer, child CSPRNGs can cheaply detect this condition and reseed. To do so, they just need to track an additional 64-bit value in the associated state, and compare it against the root seed version (generation) on random reads. This revision integrates arc4random(9) into that model without substantially changing the design or implementation of arc4random(9). The motivation is that arc4random(9) is immediately reseeded when the backing random(4) implementation has additional entropy. This is arguably most important during boot, when fenestrasX is reseeding at 1, 3, 9, 27, etc., second intervals. Today, arc4random(9) has a hardcoded 300 second reseed window. Without this mechanism, if arc4random(9) gets weak entropy during initial seed (and arc4random(9) is used early in boot, so this is quite possible), it may continue to emit poorly seeded output for 5 minutes. The FenestrasX push-reseed scheme corrects consumers, like arc4random(9), as soon as possible. Reviewed by: markm Approved by: csprng (markm) Differential Revision: https://reviews.freebsd.org/D22838 Notes: svn path=/head/; revision=366621
* arm64: check for CRC32 support via HWCAPMitchell Horne2020-09-081-9/+3
| | | | | | | | | | | | | | Doing it this way eliminates the assumption about homogeneous support for the feature, since HWCAP values are only set if support is present on all CPUs. Reviewed by: tuexen, markj MFC after: 1 week Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D26032 Notes: svn path=/head/; revision=365461
* libkern: clean up empty lines in .c and .h filesMateusz Guzik2020-09-019-28/+0
| | | | Notes: svn path=/head/; revision=365081
* Smaller crc for the boot loader.Warner Losh2020-09-011-0/+8
| | | | | | | | | | | | | | | | | Save 7k of text space by using simpler crc32 for standalone case. we don't need all that fancy optimization in the boot loader, so use a simplified version of the CRC function. We could save more by doing it one bit at a time rather than 32, but this is the biggest savings at the smallest performance hit. With LUA and verfied exec, gptboot, gptzfsboot and friends are pushing the ~530k limit and every little bit helps. Reviewed By: allanjude Differential Revision: https://reviews.freebsd.org/D24225 Notes: svn path=/head/; revision=365022
* Add prng(9) APIConrad Meyer2020-08-131-32/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Add prng(9) as a replacement for random(9) in the kernel. There are two major differences from random(9) and random(3): - General prng(9) APIs (prng32(9), etc) do not guarantee an implementation or particular sequence; they should not be used for repeatable simulations. - However, specific named API families are also exposed (for now: PCG), and those are expected to be repeatable (when so-guaranteed by the named algorithm). Some minor differences from random(3) and earlier random(9): - PRNG state for the general prng(9) APIs is per-CPU; this eliminates contention on PRNG state in SMP workloads. Each PCPU generator in an SMP system produces a unique sequence. - Better statistical properties than the Park-Miller ("minstd") PRNG (longer period, uniform distribution in all bits, passes BigCrush/PractRand analysis). - Faster than Park-Miller ("minstd") PRNG -- no division is required to step PCG-family PRNGs. For now, random(9) becomes a thin shim around prng32(). Eventually I would like to mechanically switch consumers over to the explicit API. Reviewed by: kib, markj (previous version both) Discussed with: markm Differential Revision: https://reviews.freebsd.org/D25916 Notes: svn path=/head/; revision=364219
* libkern: Add arc4random_uniformEmmanuel Vadot2020-05-231-0/+58
| | | | | | | | | | | | This variant get a random number up to the limit passed as the argument. This is simply a copy of the libc version. Sponsored-by: The FreeBSD Foundation Reviewed by: cem, hselasky (previous version) Differential Revision: https://reviews.freebsd.org/D24962 Notes: svn path=/head/; revision=361418
* Mark more nodes as CTLFLAG_MPSAFE or CTLFLAG_NEEDGIANT (17 of many)Pawel Biernacki2020-02-261-7/+14
| | | | | | | | | | | | | | | | | | | r357614 added CTLFLAG_NEEDGIANT to make it easier to find nodes that are still not MPSAFE (or already are but aren’t properly marked). Use it in preparation for a general review of all nodes. This is non-functional change that adds annotations to SYSCTL_NODE and SYSCTL_PROC nodes using one of the soon-to-be-required flags. Mark all obvious cases as MPSAFE. All entries that haven't been marked as MPSAFE before are by default marked as NEEDGIANT Approved by: kib (mentor, blanket) Commented by: kib, gallatin, melifaro Differential Revision: https://reviews.freebsd.org/D23718 Notes: svn path=/head/; revision=358333
* random(9): Deprecate random(9), remove meaningless srandom(9)Conrad Meyer2019-12-261-13/+12
| | | | | | | | | | | | | | | | | | | | | | | | srandom(9) is meaningless on SMP systems or any system with, say, interrupts. One could never rely on random(9) to produce a reproducible sequence of outputs on the basis of a specific srandom() seed because the global state was shared by all kernel contexts. As such, removing it is literally indistinguishable to random(9) consumers (as compared with retaining it). Mark random(9) as deprecated and slated for quick removal. This is not to say we intend to remove all fast, non-cryptographic PRNG(s) in the kernel. It/they just won't be random(9), as it exists today, in either name or implementation. Before random(9) is removed, a replacement will be provided and in-tree consumers will be converted. Note that despite the name, the random(9) interface does not bear any resemblance to random(3). Instead, it is the same crummy 1988 Park-Miller LCG used in libc rand(3). Notes: svn path=/head/; revision=356097
* Port the NetBSD KCSAN runtime to FreeBSD.Andrew Turner2019-11-213-3/+3
| | | | | | | | | | | | | | | | Update the NetBSD Kernel Concurrency Sanitizer (KCSAN) runtime to work in the FreeBSD kernel. It is a useful tool for finding data races between threads executing on different CPUs. This can be enabled by enabling KCSAN in the kernel config, or by using the GENERIC-KCSAN amd64 kernel. It works on amd64 and arm64, however the later needs a compiler change to allow -fsanitize=thread that KCSAN uses. Sponsored by: DARPA, AFRL Differential Revision: https://reviews.freebsd.org/D22315 Notes: svn path=/head/; revision=354942
* Rename the macros to extract a single arm64 ID field.Andrew Turner2019-10-301-1/+1
| | | | | | | | | | | Because of the previous naming scheme the old ID_AA64PFR0_EL1 macro collided with a potential macro for the register of the same name. To fix this collision rename these macros. Sponsored by: DARPA, AFRL Notes: svn path=/head/; revision=354170
* Remove zlib 1.0.4 from kernel.Xin LI2019-08-251-5388/+0
| | | | | | | | | PR: 229763 Reviewed by: emaste, Yoshihiro Ota <ota j email ne jp> Differential Revision: https://reviews.freebsd.org/D21375 Notes: svn path=/head/; revision=351480
* libkern: Implement strchrnul(3)Conrad Meyer2019-08-191-0/+41
| | | | Notes: svn path=/head/; revision=351237
* Allow Kernel to link in both legacy libkern/zlib and new sys/contrib/zlib,Xin LI2019-08-011-34/+8
| | | | | | | | | | | | | | | | | | | | | | | | with an eventual goal to convert all legacl zlib callers to the new zlib version: * Move generic zlib shims that are not specific to zlib 1.0.4 to sys/dev/zlib. * Connect new zlib (1.2.11) to the zlib kernel module, currently built with Z_SOLO. * Prefix the legacy zlib (1.0.4) with 'zlib104_' namespace. * Convert sys/opencrypto/cryptodeflate.c to use new zlib. * Remove bundled zlib 1.2.3 from ZFS and adapt it to new zlib and make it depend on the zlib module. * Fix Z_SOLO build of new zlib. PR: 229763 Submitted by: Yoshihiro Ota <ota j email ne jp> Reviewed by: markm (sys/dev/zlib/zlib_kmod.c) Relnotes: yes Differential Revision: https://reviews.freebsd.org/D19706 Notes: svn path=/head/; revision=350496
* Remove pre-FreeBSD 7.0 compatibility.Warner Losh2019-07-191-4/+0
| | | | Notes: svn path=/head/; revision=350145
* Separate kernel crc32() implementation to its own header (gsb_crc32.h) andXin LI2019-06-172-6/+4
| | | | | | | | | | | | | rename the source to gsb_crc32.c. This is a prerequisite of unifying kernel zlib instances. PR: 229763 Submitted by: Yoshihiro Ota <ota at j.email.ne.jp> Differential Revision: https://reviews.freebsd.org/D20193 Notes: svn path=/head/; revision=349151
* random(4): depessimize arc4randomMateusz Guzik2019-05-121-3/+5
| | | | | | | | | | | | | | | - __predict_false reseeding on entry as it is almost never true. - don't blindly atomic_cmpset as on x86 it ends up dirtying the cacheline. it almost ever succeeds per above - fetch the timestamp prior to getting the cpu number Reviewed by: cem Approved by: secteam (delphij) Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D20242 Notes: svn path=/head/; revision=347500
* Revert r346410 and r346411Conrad Meyer2019-04-191-62/+0
| | | | | | | | libkern in .PATH has too many filename conflicts with libc and my -DNO_CLEAN tinderbox didn't catch that ahead of time. Mea culpa. Notes: svn path=/head/; revision=346420
* libkern: Bring in arc4random_uniform(9) from libcConrad Meyer2019-04-191-0/+62
| | | | | | | | | | | | | | | | It is a useful arc4random wrapper in the kernel for much the same reasons as in userspace. Move the source to libkern (because kernel build is restricted to sys/, but userspace can include any file it likes) and build kernel and libc versions from the same source file. Copy the documentation from arc4random_uniform(3) to the section 9 page. While here, add missing arc4random_buf(9) symlink. Sponsored by: Dell EMC Isilon Notes: svn path=/head/; revision=346410
* random(4): Restore availability tradeoff prior to r346250Conrad Meyer2019-04-181-6/+40
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | As discussed in that commit message, it is a dangerous default. But the safe default causes enough pain on a variety of platforms that for now, restore the prior default. Some of this is self-induced pain we should/could do better about; for example, programmatic CI systems and VM managers should introduce entropy from the host for individual VM instances. This is considered a future work item. On modern x86 and Power9 systems, this may be wholly unnecessary after D19928 lands (even in the non-ideal case where early /boot/entropy is unavailable), because they have fast hardware random sources available early in boot. But D19928 is not yet landed and we have a host of architectures which do not provide fast random sources. This change adds several tunables and diagnostic sysctls, documented thoroughly in UPDATING and sys/dev/random/random_infra.c. PR: 230875 (reopens) Reported by: adrian, jhb, imp, and probably others Reviewed by: delphij, imp (earlier version), markm (earlier version) Discussed with: adrian Approved by: secteam(delphij) Relnotes: yeah Security: related Differential Revision: https://reviews.freebsd.org/D19944 Notes: svn path=/head/; revision=346358
* random(4): Block read_random(9) on initial seedingConrad Meyer2019-04-151-26/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | read_random() is/was used, mostly without error checking, in a lot of very sensitive places in the kernel -- including seeding the widely used arc4random(9). Most uses, especially arc4random(9), should block until the device is seeded rather than proceeding with a bogus or empty seed. I did not spy any obvious kernel consumers where blocking would be inappropriate (in the sense that lack of entropy would be ok -- I did not investigate locking angle thoroughly). In many instances, arc4random_buf(9) or that family of APIs would be more appropriate anyway; that work was done in r345865. A minor cleanup was made to the implementation of the READ_RANDOM function: instead of using a variable-length array on the stack to temporarily store all full random blocks sufficient to satisfy the requested 'len', only store a single block on the stack. This has some benefit in terms of reducing stack usage, reducing memcpy overhead and reducing devrandom output leakage via the stack. Additionally, the stack block is now safely zeroed if it was used. One caveat of this change is that the kern.arandom sysctl no longer returns zero bytes immediately if the random device is not seeded. This means that FreeBSD-specific userspace applications which attempted to handle an unseeded random device may be broken by this change. If such behavior is needed, it can be replaced by the more portable getrandom(2) GRND_NONBLOCK option. On any typical FreeBSD system, entropy is persisted on read/write media and used to seed the random device very early in boot, and blocking is never a problem. This change primarily impacts the behavior of /dev/random on embedded systems with read-only media that do not configure "nodevice random". We toggle the default from 'charge on blindly with no entropy' to 'block indefinitely.' This default is safer, but may cause frustration. Embedded system designers using FreeBSD have several options. The most obvious is to plan to have a small writable NVRAM or NAND to persist entropy, like larger systems. Early entropy can be fed from any loader, or by writing directly to /dev/random during boot. Some embedded SoCs now provide a fast hardware entropy source; this would also work for quickly seeding Fortuna. A 3rd option would be creating an embedded-specific, more simplistic random module, like that designed by DJB in [1] (this design still requires a small rewritable media for forward secrecy). Finally, the least preferred option might be "nodevice random", although I plan to remove this in a subsequent revision. To help developers emulate the behavior of these embedded systems on ordinary workstations, the tunable kern.random.block_seeded_status was added. When set to 1, it blocks the random device. I attempted to document this change in random.4 and random.9 and ran into a bunch of out-of-date or irrelevant or inaccurate content and ended up rototilling those documents more than I intended to. Sorry. I think they're in a better state now. PR: 230875 Reviewed by: delphij, markm (earlier version) Approved by: secteam(delphij), devrandom(markm) Relnotes: yes Differential Revision: https://reviews.freebsd.org/D19744 Notes: svn path=/head/; revision=346250
* Add non-sleepable strdup variant strdup_flagsMatt Macy2019-02-201-2/+11
| | | | | | | | | | | | debugfs expects to do non-sleepable allocations Reviewed by: hps@ MFC after: 1 week Sponsored by: iX Systems Differential Revision: https://reviews.freebsd.org/D19259 Notes: svn path=/head/; revision=344384
* mcount: tidy up ANSIficationEd Maste2018-10-201-4/+5
| | | | | | | | | | | | libc/gmon's mcount was ANSIfied in r124180, with libkern following over a decade later, in r325988, but some minor discrepancies remained. Update libc/gmon's mexitcount to an ANSI C function definition, and use (void) for libkern-only functions that take no arguments. Reported by: bde Notes: svn path=/head/; revision=339505
* Sync strlcpy with userland version, againConrad Meyer2018-06-211-20/+19
| | | | | | | | | | | | No functional change. Please remember to update libkern copies of libc functions when you update libc. Sponsored by: Dell EMC Isilon Notes: svn path=/head/; revision=335498
* str(r)chr: Replace union abuse with __DECONSTConrad Meyer2018-06-042-23/+16
| | | | | | | | | | | | | Writing one union member and reading another is technically illegal C, although we do it in many places in the tree. Use the __DECONST macro instead, which is (technically) a valid C construct. Trivial style(9) cleanups to touched lines while here. Sponsored by: Dell EMC Isilon Notes: svn path=/head/; revision=334624
* Use __builtin for various mem* and b* (e.g. bzero) routines.Mateusz Guzik2018-06-023-3/+3
| | | | | | | | | | | | | | | | | Some of the routines were using artificially limited builtin already, drop the explicit limit. The use of builtins allows quite often allows the compiler to elide the call or most zeroing to begin with. For instance, if the target object is 32 bytes in size and gets zeroed + has 16 bytes initialized, the compiler can just add code to zero out the rest. Note not all the primites have asm variants and some of the existing ones are not optimized. Maintaines are strongly encourage to take a look (regardless of this change). Notes: svn path=/head/; revision=334534
* libkern: tidy up memsetMateusz Guzik2018-06-021-5/+2
| | | | | | | | | | | | | | 1. Remove special-casing of 0 as it just results in an extra function call. This is clearly pessimal. 2. Drop the inline stuff. For the most part it is much better served with __builtin_memset (coming later). 3. Move the declaration to systm.h to match other funcs. Archs are encouraged to implement the variant for their own platform so that this implementation can be dropped. Notes: svn path=/head/; revision=334533
* Fix recent breakages of kernel profiling, mostly on i386 (high resolutionBruce Evans2018-06-021-0/+18
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | kernel profiling remains broken). memmove() was broken using ALTENTRY(). ALTENTRY() is only different from ENTRY() in the profiling case, and its use in that case was sort of backwards. The backwardness magically turned memmove() into memcpy() instead of completely breaking it. Only the high resolution parts of profiling itself were broken. Use ordinary ENTRY() for memmove(). Turn bcopy() into a tail call to memmove() to reduce complications. This gives slightly different pessimizations and profiling lossage. The pessimizations are minimized by not using a frame pointer() for bcopy(). Calls to profiling functions from exception trampolines were not relocated. This caused crashes on the first exception. Fix this using function pointers. Addresses of exception handlers in trampolines were not relocated. This caused unknown offsets in the profiling data. Relocate by abusing setidt_disp as for pmc although this is slower than necessary and requires namespace pollution. pmc seems to be missing some relocations. Stack traces and lots of other things in debuggers need similar relocations. Most user addresses were misclassified as unknown kernel addresses and then ignored. Treat all unknown addresses as user. Now only user addresses in the kernel text range are significantly misclassified (as known kernel addresses). The ibrs functions didn't preserve enough registers. This is the only recent breakage on amd64. Although these functions are written in asm, in the profiling case they call profiling functions which are mostly for the C ABI, so they only have to save call-used registers. They also have to save arg and return registers in some cases and actually save them in all cases to reduce complications. They end up saving all registers except %ecx on i386 and %r10 and %r11 on amd64. Saving these is only needed for 1 caller on each of amd64 and i386. Save them there. This is slightly simpler. Remove saving %ecx in handle_ibrs_exit on i386. Both handle_ibrs_entry and handle_ibrs_exit use %ecx, but only the latter needed to or did save it. But saving it there doesn't work for the profiling case. amd64 has more automatic saving of the most common scratch registers %rax, %rcx and %rdx (its complications for %r10 are from unusual use of %r10 by SYSCALL). Thus profiling of handle_ibrs_exit_rs() was not broken, and I didn't simplify the saving by moving the saving of these registers from it to the caller. Notes: svn path=/head/; revision=334520
* Protect bzero call against macro expansionWarner Losh2018-05-241-1/+1
| | | | | | | | | | | Shortly, we'll be moving to defining bzero and memset in terms of __builting_memset. To do that, we can't have macro calls to bzero in the fallback impelmentation of memset. Normal calls to bzero are fine. All 4 architectures that use this have their own copies of bzero, so there's no mutual recursion issue between memset and bcopy. Notes: svn path=/head/; revision=334192
* This is no unreferenced, so retire it.Warner Losh2018-05-241-40/+0
| | | | | | | Differential Revision: https://reviews.freebsd.org/D15374 Notes: svn path=/head/; revision=334184
generated by cgit v1.2.3 (git 2.25.1) at 2024-09-25 13:56:23 +0000