aboutsummaryrefslogtreecommitdiff
path: root/sys/kern/kern_mutex.c
Commit message (Collapse)AuthorAgeFilesLines
...
* locks: remove SCHEDULER_STOPPED checks from primitives for modulesMateusz Guzik2017-02-171-6/+0
| | | | | | | | | | | They all fallback to the slow path if necessary and the check is there. This means a panicked kernel executing code from modules will be able to succeed doing actual lock/unlock, but this was already the case for core code which has said primitives inlined. Notes: svn path=/head/; revision=313853
* locks: tidy up unlock fallback pathsMateusz Guzik2017-02-091-7/+10
| | | | | | | | | | | Update comments to note these functions are reachable if lockstat is enabled. Check if the lock has any bits set before attempting unlock, which saves an unnecessary atomic operation. Notes: svn path=/head/; revision=313467
* locks: change backoff to exponentialMateusz Guzik2017-02-071-45/+9
| | | | | | | | | | | | | | | Previous implementation would use a random factor to spread readers and reduce chances of starvation. This visibly reduces effectiveness of the mechanism. Switch to the more traditional exponential variant. Try to limit starvation by imposing an upper limit of spins after which spinning is half of what other threads get. Note the mechanism is turned off by default. Reviewed by: kib (previous version) Notes: svn path=/head/; revision=313386
* locks: fix recursion support after recent changesMateusz Guzik2017-02-061-0/+2
| | | | | | | | | | | | When a relevant lockstat probe is enabled the fallback primitive is called with a constant signifying a free lock. This works fine for typical cases but breaks with recursion, since it checks if the passed value is that of the executing thread. Read the value if necessary. Notes: svn path=/head/; revision=313335
* mtx: fixup r313278, the assignemnt was supposed to go inside the loopMateusz Guzik2017-02-051-1/+1
| | | | Notes: svn path=/head/; revision=313279
* mtx: fix up _mtx_obtain_lock_fetch usage in thread lockMateusz Guzik2017-02-051-0/+1
| | | | | | | | Since _mtx_obtain_lock_fetch no longer sets the argument to MTX_UNOWNED, callers have to do it on their own. Notes: svn path=/head/; revision=313278
* mtx: move lockstat handling out of inline primitivesMateusz Guzik2017-02-051-8/+12
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Lockstat requires checking if it is enabled and if so, calling a 6 argument function. Further, determining whether to call it on unlock requires pre-reading the lock value. This is problematic in at least 3 ways: - more branches in the hot path than necessary - additional cacheline ping pong under contention - bigger code Instead, check first if lockstat handling is necessary and if so, just fall back to regular locking routines. For this purpose a new macro is introduced (LOCKSTAT_PROFILE_ENABLED). LOCK_PROFILING uninlines all primitives. Fold in the current inline lock variant into the _mtx_lock_flags to retain the support. With this change the inline variants are not used when LOCK_PROFILING is defined and thus can ignore its existence. This results in: text data bss dec hex filename 22259667 1303208 4994976 28557851 1b3c21b kernel.orig 21797315 1303208 4994976 28095499 1acb40b kernel.patched i.e. about 3% reduction in text size. A remaining action is to remove spurious arguments for internal kernel consumers. Notes: svn path=/head/; revision=313275
* mtx: switch to fcmpsetMateusz Guzik2017-02-051-17/+15
| | | | | | | | | | | | | | | The found value is passed to locking routines in order to reduce cacheline accesses. mtx_unlock grows an explicit check for regular unlock. On ll/sc architectures the routine can fail even if the lock could have been handled by the inline primitive. Discussed with: jhb Tested by: pho (previous version) Notes: svn path=/head/; revision=313269
* Sprinkle __read_mostly on backoff and lock profiling code.Mateusz Guzik2017-01-271-2/+2
| | | | | | | MFC after: 1 month Notes: svn path=/head/; revision=312890
* mtx: plug open-coded mtx_lock access missed in r311172Mateusz Guzik2017-01-041-1/+1
| | | | Notes: svn path=/head/; revision=311226
* Reduce lock accesses in thread lock similarly to r311172.Mateusz Guzik2017-01-031-6/+12
| | | | Notes: svn path=/head/; revision=311194
* mtx: reduce lock accessesMateusz Guzik2017-01-031-39/+50
| | | | | | | | | | | | | | | | | Instead of spuriously re-reading the lock value, read it once. This change also has a side effect of fixing a performance bug: on failed _mtx_obtain_lock, it was possible that re-read would find the lock is unowned, but in this case the primitive would make a trip through turnstile code. This is diff reduction to a variant which uses atomic_fcmpset. Discussed with: jhb (previous version) Tested by: pho (previous version) Notes: svn path=/head/; revision=311172
* Use a consistent snapshot of the lock state in owner_mtx().Mark Johnston2016-12-101-3/+6
| | | | | | | MFC after: 2 weeks Notes: svn path=/head/; revision=309784
* Make no assertions about mutex state when the scheduler is stopped.Eric van Gyzen2016-09-261-1/+1
| | | | | | | | | | This changes the assert path to match the lock and unlock paths. MFC after: 1 week Sponsored by: Dell EMC Notes: svn path=/head/; revision=306346
* locks: add backoff for spin mutexes and thread lockMateusz Guzik2016-09-091-13/+50
| | | | | | | Reviewed by: jhb Notes: svn path=/head/; revision=305671
* locks: fix compilation for KDTRACE_HOOKS && !ADAPTIVE_* caseMateusz Guzik2016-08-021-1/+3
| | | | | | | Reported by: Michael Butler <imb protected-networks.net> Notes: svn path=/head/; revision=303655
* Implement trivial backoff for locking primitives.Mateusz Guzik2016-08-011-9/+42
| | | | | | | | | | | | | | | | | | | | | | | | All current spinning loops retry an atomic op the first chance they get, which leads to performance degradation under load. One classic solution to the problem consists of delaying the test to an extent. This implementation has a trivial linear increment and a random factor for each attempt. For simplicity, this first thouch implementation only modifies spinning loops where the lock owner is running. spin mutexes and thread lock were not modified. Current parameters are autotuned on boot based on mp_cpus. Autotune factors are very conservative and are subject to change later. Reviewed by: kib, jhb Tested by: pho MFC after: 1 week Notes: svn path=/head/; revision=303643
* locks: change sleep_cnt and spin_cnt types to u_intMateusz Guzik2016-07-311-2/+2
| | | | | | | | | | | Both variables are uint64_t, but they only count spins or sleeps. All reasonable values which we can get here comfortably hit in 32-bit range. Suggested by: kib MFC after: 1 week Notes: svn path=/head/; revision=303584
* Implement mtx_trylock_spin(9).Konstantin Belousov2016-07-231-0/+28
| | | | | | | | | | | Discussed with: bde Reviewed by: jhb Sponsored by: The FreeBSD Foundation MFC after: 1 week Differential revision: https://reviews.freebsd.org/D7192 Notes: svn path=/head/; revision=303211
* Ensure that spinlock sections are balanced even after a panic.Mark Johnston2016-07-051-1/+8
| | | | | | | | | | | | | | | vpanic() uses spinlock_enter() to disable interrupts before dumping core. However, when the scheduler is stopped and INVARIANTS is not configured, thread_lock() does not acquire a spinlock section, while thread_unlock() releases one. This can result in interrupts staying enabled while the kernel dumps core, complicating post-mortem analysis of the crash. Approved by: re (gjb) MFC after: 1 week Sponsored by: EMC / Isilon Storage Division Notes: svn path=/head/; revision=302346
* Microoptimize locking primitives by avoiding unnecessary atomic ops.Mateusz Guzik2016-06-011-4/+9
| | | | | | | | | | | | | Inline version of primitives do an atomic op and if it fails they fallback to actual primitives, which immediately retry the atomic op. The obvious optimisation is to check if the lock is free and only then proceed to do an atomic op. Reviewed by: jhb, vangyzen Notes: svn path=/head/; revision=301157
* Remove the MUTEX_DEBUG kernel option.Mark Johnston2016-05-181-36/+0
| | | | | | | | | It has no counterpart among the other lock primitives and has been a no-op for years. Mutex consistency checks are generally done whenver INVARIANTS is enabled. Notes: svn path=/head/; revision=300106
* Guard the lockstat:::thread-spin probe with KDTRACE_HOOKS.Mark Johnston2016-05-181-0/+2
| | | | | | | X-MFC-With: r300103 Notes: svn path=/head/; revision=300104
* lockstat:::thread-spin should only fire after spinning for the lock.Mark Johnston2016-05-181-1/+2
| | | | | | | MFC after: 1 week Notes: svn path=/head/; revision=300103
* Don't modify curthread->td_locks unless INVARIANTS is enabled.Mark Johnston2015-08-021-4/+4
| | | | | | | | | | | | | This field is only used in a KASSERT that verifies that no locks are held when returning to user mode. Moreover, the td_locks accounting is only correct when LOCK_DEBUG > 0, which is implied by INVARIANTS. Reviewed by: jhb MFC after: 1 week Differential Revision: https://reviews.freebsd.org/D3205 Notes: svn path=/head/; revision=286166
* Implement the lockstat provider using SDT(9) instead of the custom providerMark Johnston2015-07-191-10/+10
| | | | | | | | | | | in lockstat.ko. This means that lockstat probes now have typed arguments and will utilize SDT probe hot-patching support when it arrives. Reviewed by: gnn Differential Revision: https://reviews.freebsd.org/D2993 Notes: svn path=/head/; revision=285703
* Fix the !KDTRACE_HOOKS build.Mark Johnston2015-07-181-0/+2
| | | | | | | X-MFC-With: r285664 Notes: svn path=/head/; revision=285667
* Pass the lock object to lockstat_nsecs() and return immediately ifMark Johnston2015-07-181-9/+10
| | | | | | | | | | | | LO_NOPROFILE is set. Some timecounter handlers acquire a spin mutex, and we don't want to recurse if lockstat probes are enabled. PR: 201642 Reviewed by: avg MFC after: 3 days Notes: svn path=/head/; revision=285664
* several lockstat improvementsAndriy Gapon2015-06-121-9/+25
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 0. For spin events report time spent spinning, not a loop count. While loop count is much easier and cheaper to obtain it is hard to reason about the reported numbers, espcially for adaptive locks where both spinning and sleeping can happen. So, it's better to compare apples and apples. 1. Teach lockstat about FreeBSD rw locks. This is done in part by changing the corresponding probes and in part by changing what probes lockstat should expect. 2. Teach lockstat that rw locks are adaptive and can spin on FreeBSD. 3. Report lock acquisition events for successful rw try-lock operations. 4. Teach lockstat about FreeBSD sx locks. Reporting of events for those locks completely mirrors rw locks. 5. Report spin and block events before acquisition event. This is behavior documented for the upstream, so it makes sense to stick to it. Note that because of FreeBSD adaptive lock implementations both the spin and block events may be reported for the same acquisition while the upstream reports only one of them. Differential Revision: https://reviews.freebsd.org/D2727 Reviewed by: markj MFC after: 17 days Relnotes: yes Sponsored by: ClusterHQ Notes: svn path=/head/; revision=284297
* Add _NEW flag to mtx(9), sx(9), rmlock(9) and rwlock(9).Dmitry Chagin2014-12-131-1/+3
| | | | | | | | | | | A _NEW flag passed to _init_flags() to avoid check for double-init. Differential Revision: https://reviews.freebsd.org/D1208 Reviewed by: jhb, wblock MFC after: 1 Month Notes: svn path=/head/; revision=275751
* Disable recursion for the process spinlock.Konstantin Belousov2014-12-011-1/+1
| | | | | | | | | | Tested by: pho Discussed with: jhb Sponsored by: The FreeBSD Foundation MFC after: 1 month Notes: svn path=/head/; revision=275372
* The process spin lock currently has the following distinct uses:Konstantin Belousov2014-11-261-0/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - Threads lifetime cycle, in particular, counting of the threads in the process, and interlocking with process mutex and thread lock. The main reason of this is that turnstile locks are after thread locks, so you e.g. cannot unlock blockable mutex (think process mutex) while owning thread lock. - Virtual and profiling itimers, since the timers activation is done from the clock interrupt context. Replace the p_slock by p_itimmtx and PROC_ITIMLOCK(). - Profiling code (profil(2)), for similar reason. Replace the p_slock by p_profmtx and PROC_PROFLOCK(). - Resource usage accounting. Need for the spinlock there is subtle, my understanding is that spinlock blocks context switching for the current thread, which prevents td_runtime and similar fields from changing (updates are done at the mi_switch()). Replace the p_slock by p_statmtx and PROC_STATLOCK(). The split is done mostly for code clarity, and should not affect scalability. Tested by: pho Sponsored by: The FreeBSD Foundation MFC after: 1 week Notes: svn path=/head/; revision=275121
* opt_global.h is included automatically in the build. No need toWarner Losh2014-11-181-1/+0
| | | | | | | | | explicitly include it in these places. Sponsored by: Netflix Notes: svn path=/head/; revision=274668
* Add a new thread state "spinning" to schedgraph and add tracepoints at theJohn Baldwin2014-11-041-0/+11
| | | | | | | start and stop of spinning waits in lock primitives. Notes: svn path=/head/; revision=274092
* - For kernel compiled only with KDTRACE_HOOKS and not any lock debuggingAttilio Rao2013-11-251-3/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | option, unbreak the lock tracing release semantic by embedding calls to LOCKSTAT_PROFILE_RELEASE_LOCK() direclty in the inlined version of the releasing functions for mutex, rwlock and sxlock. Failing to do so skips the lockstat_probe_func invokation for unlocking. - As part of the LOCKSTAT support is inlined in mutex operation, for kernel compiled without lock debugging options, potentially every consumer must be compiled including opt_kdtrace.h. Fix this by moving KDTRACE_HOOKS into opt_global.h and remove the dependency by opt_kdtrace.h for all files, as now only KDTRACE_FRAMES is linked there and it is only used as a compile-time stub [0]. [0] immediately shows some new bug as DTRACE-derived support for debug in sfxge is broken and it was never really tested. As it was not including correctly opt_kdtrace.h before it was never enabled so it was kept broken for a while. Fix this by using a protection stub, leaving sfxge driver authors the responsibility for fixing it appropriately [1]. Sponsored by: EMC / Isilon storage division Discussed with: rstone [0] Reported by: rstone [1] Discussed with: philip Notes: svn path=/head/; revision=258541
* Fix lc_lock/lc_unlock() support for rmlocks held in shared mode. WithDavide Italiano2013-09-201-8/+8
| | | | | | | | | | | | | | | | | | current lock classes KPI it was really difficult because there was no way to pass an rmtracker object to the lock/unlock routines. In order to accomplish the task, modify the aforementioned functions so that they can return (or pass as argument) an uinptr_t, which is in the rm case used to hold a pointer to struct rm_priotracker for current thread. As an added bonus, this fixes rm_sleep() in the rm shared case, which right now can communicate priotracker structure between lc_unlock()/lc_lock(). Suggested by: jhb Reviewed by: jhb Approved by: re (delphij) Notes: svn path=/head/; revision=255745
* Give mutex(9) the ability to recurse on a per-instance basis.Attilio Rao2013-08-091-6/+14
| | | | | | | | | | | | | Now the MTX_RECURSE flag can be passed to the mtx_*_flag() calls. This helps in cases we want to narrow down to specific calls the possibility to recurse for some locks. Sponsored by: EMC / Isilon storage division Reviewed by: jeff, alc Tested by: pho Notes: svn path=/head/; revision=254139
* Fix r253823. Some WIP patches snuck in.Scott Long2013-07-301-14/+6
| | | | | | | Submitted by: zont Notes: svn path=/head/; revision=253824
* Create a knob, kern.ipc.sfreadahead, that allows one to tune the amount ofScott Long2013-07-301-6/+14
| | | | | | | | | | readahead that sendfile() will do. Default remains the same. Obtained from: Netflix MFC after: 3 days Notes: svn path=/head/; revision=253823
* A few mostly cosmetic nits to aid in debugging:John Baldwin2013-06-251-3/+3
| | | | | | | | | | | - Call lock_init() first before setting any lock_object fields in lock init routines. This way if the machine panics due to a duplicate init the lock's original state is preserved. - Somewhat similarly, don't decrement td_locks and td_slocks until after an unlock operation has completed successfully. Notes: svn path=/head/; revision=252212
* Fixup r240424: On entering KDB backends, the hijacked thread to runAttilio Rao2012-12-221-2/+2
| | | | | | | | | | | | | | | | interrupt context can still be idlethread. At that point, without the panic condition, it can still happen that idlethread then will try to acquire some locks to carry on some operations. Skip the idlethread check on block/sleep lock operations when KDB is active. Reported by: jh Tested by: jh MFC after: 1 week Notes: svn path=/head/; revision=244582
* Give mtx(9) the ability to crunch different type of structures, with theAttilio Rao2012-10-311-17/+64
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | only constraint that they have a lock cookie named mtx_lock. This name, then, becames reserved from the struct that wants to use the mtx(9) KPI and other locking primitives cannot reuse it for their members. Namely such structs are the current struct mtx and the new struct mtx_padalign. The new structure will define an object which is the same as the same layout of a struct mtx but will be allocated in areas aligned to the cache line size and will be as big as a cache line. This is supposed to give higher performance for highly contented mutexes both spin or sleep (because of the adaptive spinning), where the cache line contention results in too much traffic on the system bus. The struct mtx_padalign can be used in a completely transparent way with the mtx(9) KPI. At the moment, a possibility to MFC the patch should be carefully evaluated because this patch breaks the low level KPI (not its representation though). Discussed with: jhb Reviewed by: jeff, andre Reviewed by: mdf (earlier version) Tested by: jimharris Notes: svn path=/head/; revision=242395
* Remove all the checks on curthread != NULL with the exception of some MDAttilio Rao2012-09-131-5/+0
| | | | | | | | | | | | | | trap checks (eg. printtrap()). Generally this check is not needed anymore, as there is not a legitimate case where curthread != NULL, after pcpu 0 area has been properly initialized. Reviewed by: bde, jhb MFC after: 1 week Notes: svn path=/head/; revision=240475
* Improve check coverage about idle threads.Attilio Rao2012-09-121-0/+6
| | | | | | | | | | | | | | | Idle threads are not allowed to acquire any lock but spinlocks. Deny any attempt to do so by panicing at the locking operation when INVARIANTS is on. Then, remove the check on blocking on a turnstile. The check in sleepqueues is left because they are not allowed to use tsleep() either which could happen still. Reviewed by: bde, jhb, kib MFC after: 1 week Notes: svn path=/head/; revision=240424
* Add software PMC support.Fabien Thomas2012-03-281-0/+15
| | | | | | | | | | | | | | | | New kernel events can be added at various location for sampling or counting. This will for example allow easy system profiling whatever the processor is with known tools like pmcstat(8). Simultaneous usage of software PMC and hardware PMC is possible, for example looking at the lock acquire failure, page fault while sampling on instructions. Sponsored by: NETASQ MFC after: 1 month Notes: svn path=/head/; revision=233628
* panic: add a switch and infrastructure for stopping other CPUs in SMP caseAndriy Gapon2011-12-111-0/+25
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Historical behavior of letting other CPUs merily go on is a default for time being. The new behavior can be switched on via kern.stop_scheduler_on_panic tunable and sysctl. Stopping of the CPUs has (at least) the following benefits: - more of the system state at panic time is preserved intact - threads and interrupts do not interfere with dumping of the system state Only one thread runs uninterrupted after panic if stop_scheduler_on_panic is set. That thread might call code that is also used in normal context and that code might use locks to prevent concurrent execution of certain parts. Those locks might be held by the stopped threads and would never be released. To work around this issue, it was decided that instead of explicit checks for panic context, we would rather put those checks inside the locking primitives. This change has substantial portions written and re-written by attilio and kib at various times. Other changes are heavily based on the ideas and patches submitted by jhb and mdf. bde has provided many insights into the details and history of the current code. The new behavior may cause problems for systems that use a USB keyboard for interfacing with system console. This is because of some unusual locking patterns in the ukbd code which have to be used because on one hand ukbd is below syscons, but on the other hand it has to interface with other usb code that uses regular mutexes/Giant for its concurrency protection. Dumping to USB-connected disks may also be affected. PR: amd64/139614 (at least) In cooperation with: attilio, jhb, kib, mdf Discussed with: arch@, bde Tested by: Eugene Grosbein <eugen@grosbein.net>, gnn, Steven Hartland <killing@multiplay.co.uk>, glebius, Andrew Boyer <aboyer@averesystems.com> (various versions of the patch) MFC after: 3 months (or never) Notes: svn path=/head/; revision=228424
* Introduce macro stubs in the mutex implementation that will be alwaysAttilio Rao2011-11-201-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | defined and will allow consumers, willing to provide options, file and line to locking requests, to not worry about options redefining the interfaces. This is typically useful when there is the need to build another locking interface on top of the mutex one. The introduced functions that consumers can use are: - mtx_lock_flags_ - mtx_unlock_flags_ - mtx_lock_spin_flags_ - mtx_unlock_spin_flags_ - mtx_assert_ - thread_lock_flags_ Spare notes: - Likely we can get rid of all the 'INVARIANTS' specification in the ppbus code by using the same macro as done in this patch (but this is left to the ppbus maintainer) - all the other locking interfaces may require a similar cleanup, where the most notable case is sx which will allow a further cleanup of vm_map locking facilities - The patch should be fully compatible with older branches, thus a MFC is previewed (infact it uses all the underlying mechanisms already present). Comments review by: eadler, Ben Kaduk Discussed with: kib, jhb MFC after: 1 month Notes: svn path=/head/; revision=227758
* Constify arguments for locking KPIs where possible.Pawel Jakub Dawidek2011-11-161-11/+12
| | | | | | | | | | This enables locking consumers to pass their own structures around as const and be able to assert locks embedded into those structures. Reviewed by: ed, kib, jhb Notes: svn path=/head/; revision=227588
* - Remove <machine/mutex.h>. Most of the headers were empty, and theJohn Baldwin2010-11-091-11/+11
| | | | | | | | | | | | | | | contents of the ones that were not empty were stale and unused. - Now that <machine/mutex.h> no longer exists, there is no need to allow it to override various helper macros in <sys/mutex.h>. - Rename various helper macros for low-level operations on mutexes to live in the _mtx_* or __mtx_* namespaces. While here, change the names to more closely match the real API functions they are backing. - Drop support for including <sys/mutex.h> in assembly source files. Suggested by: bde (1, 2) Notes: svn path=/head/; revision=215054
* Right now, WITNESS just blindly pipes all the output to theAttilio Rao2010-05-111-1/+1
| | | | | | | | | | | | | | | | (TOCONS | TOLOG) mask even when called from DDB points. That breaks several output, where the most notable is textdump output. Fix this by having configurable callbacks passed to witness_list_locks() and witness_display_spinlock() for printing out datas. Reported by: several broken textdump outputs Tested by: Giovanni Trematerra <giovanni dot trematerra at gmail dot com> MFC after: 7 days X-MFC: r207922 Notes: svn path=/head/; revision=207929
generated by cgit v1.2.3 (git 2.25.1) at 2024-06-25 19:21:10 +0000