aboutsummaryrefslogtreecommitdiff
path: root/sys/sys/mutex.h
Commit message (Collapse)AuthorAgeFilesLines
* mtx: add mtx_wait_unlockedMateusz Guzik2020-08-041-0/+1
| | | | Notes: svn path=/head/; revision=363871
* Annotate Giant drop/pickup macros with __predict_falseMateusz Guzik2018-12-071-2/+2
| | | | | | | | | | They are used in important places of the kernel with the lock not being held majority of the time. Sponsored by: The FreeBSD Foundation Notes: svn path=/head/; revision=341683
* inline atomics and allow tied modules to inline locksMatt Macy2018-07-021-1/+1
| | | | | | | | | | | | | | - inline atomics in modules on i386 and amd64 (they were always inline on other arches) - allow modules to opt in to inlining locks by specifying MODULE_TIED=1 in the makefile Reviewed by: kib Sponsored by: Limelight Networks Differential Revision: https://reviews.freebsd.org/D16079 Notes: svn path=/head/; revision=335873
* mtx: add mtx_spin_wait_unlockedMateusz Guzik2018-02-191-0/+2
| | | | | | | | | | | | | | The primitive can be used to wait for the lock to be released. Intended usage is for locks in structures which are about to be freed. The benefit is the avoided interrupt enable/disable trip + atomic op to grab the lock and shorter wait if the lock is held (since there is no worry someone will contend on the lock, re-reads can be more aggressive). Briefly discussed with: kib Notes: svn path=/head/; revision=329540
* sys/sys: further adoption of SPDX licensing ID tags.Pedro F. Giffuni2017-11-271-0/+2
| | | | | | | | | | | | | | | Mainly focus on files that use BSD 2-Clause license, however the tool I was using misidentified many licenses so this was mostly a manual - error prone - task. The Software Package Data Exchange (SPDX) group provides a specification to make it easier for automated tools to detect and summarize well known opensource licenses. We are gradually adopting the specification, noting that the tags are considered only advisory and do not, in any way, superceed or replace the license texts. Notes: svn path=/head/; revision=326256
* Convert in-kernel thread_lock_flags calls to thread_lock when debug is disabledMateusz Guzik2017-11-251-0/+6
| | | | | | | The flags argument is not used in this case. Notes: svn path=/head/; revision=326199
* locks: pass the found lock value to unlock slow pathMateusz Guzik2017-11-221-10/+13
| | | | | | | | | This avoids an explicit read later. While here whack the cheaply obtainable 'tid' argument. Notes: svn path=/head/; revision=326107
* locks: remove the file + line argument from internal primitives when not usedMateusz Guzik2017-11-221-0/+1
| | | | | | | | | | | | | The pair is of use only in debug or LOCKPROF kernels, but was passed (zeroed) for many locks even in production kernels. While here whack the tid argument from wlock hard and xlock hard. There is no kbi change of any sort - "external" primitives still accept the pair. Notes: svn path=/head/; revision=326106
* mtx: implement thread lock fastpathMateusz Guzik2017-10-211-0/+14
| | | | | | | MFC after: 1 week Notes: svn path=/head/; revision=324836
* mtx: clean up locking spin mutexesMateusz Guzik2017-10-201-9/+16
| | | | | | | | | | | 1) shorten the fast path by pushing the lockstat probe to the slow path 2) test for kernel panic only after it turns out we will have to spin, in particular test only after we know we are not recursing MFC after: 1 week Notes: svn path=/head/; revision=324778
* mtx: change MTX_UNOWNED from 4 to 0Mateusz Guzik2017-10-131-7/+3
| | | | | | | | | | | | The value is spread all over the kernel and zeroing a register is cheaper/shorter than setting it up to an arbitrary value. Reduces amd64 GENERIC-NODEBUG .text size by 0.4%. MFC after: 1 week Notes: svn path=/head/; revision=324609
* mtx: drop the tid argument from _mtx_lock_sleepMateusz Guzik2017-09-271-8/+8
| | | | | | | | | | | tid must be equal to curthread and the target routine was already reading it anyway, which is not a problem. Not passing it as a parameter allows for a little bit shorter code in callers. MFC after: 1 week Notes: svn path=/head/; revision=324041
* locks: ensure proper barriers are used with atomic ops when necessaryMateusz Guzik2017-03-011-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Unclear how, but the locking routine for mutexes was using the *release* barrier instead of acquire. This must have been either a copy-pasto or bad completion. Going through other uses of atomics shows no barriers in: - upgrade routines (addressed in this patch) - sections protected with turnstile locks - this should be fine as necessary barriers are in the worst case provided by turnstile unlock I would like to thank Mark Millard and andreast@ for reporting the problem and testing previous patches before the issue got identified. ps. .-'---`-. ,' `. | \ | \ \ _ \ ,\ _ ,'-,/-)\ ( * \ \,' ,' ,'-) `._,) -',-') \/ ''/ ) / / / ,'-' Hardware provided by: IBM LTC Notes: svn path=/head/; revision=314474
* mtx: fix spin mutexes interaction with failed fcmpsetMateusz Guzik2017-02-201-6/+3
| | | | | | | While doing so move recursion support down to the fallback routine. Notes: svn path=/head/; revision=313996
* mtx: plug the 'opts' argument when not usedMateusz Guzik2017-02-181-5/+4
| | | | Notes: svn path=/head/; revision=313908
* mtx: restrict r313875 to kernels without LOCK_PROFILINGMateusz Guzik2017-02-171-0/+14
| | | | Notes: svn path=/head/; revision=313877
* mtx: move lockstat handling out of inline primitivesMateusz Guzik2017-02-051-13/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Lockstat requires checking if it is enabled and if so, calling a 6 argument function. Further, determining whether to call it on unlock requires pre-reading the lock value. This is problematic in at least 3 ways: - more branches in the hot path than necessary - additional cacheline ping pong under contention - bigger code Instead, check first if lockstat handling is necessary and if so, just fall back to regular locking routines. For this purpose a new macro is introduced (LOCKSTAT_PROFILE_ENABLED). LOCK_PROFILING uninlines all primitives. Fold in the current inline lock variant into the _mtx_lock_flags to retain the support. With this change the inline variants are not used when LOCK_PROFILING is defined and thus can ignore its existence. This results in: text data bss dec hex filename 22259667 1303208 4994976 28557851 1b3c21b kernel.orig 21797315 1303208 4994976 28095499 1acb40b kernel.patched i.e. about 3% reduction in text size. A remaining action is to remove spurious arguments for internal kernel consumers. Notes: svn path=/head/; revision=313275
* mtx: switch to fcmpsetMateusz Guzik2017-02-051-14/+21
| | | | | | | | | | | | | | | The found value is passed to locking routines in order to reduce cacheline accesses. mtx_unlock grows an explicit check for regular unlock. On ll/sc architectures the routine can fail even if the lock could have been handled by the inline primitive. Discussed with: jhb Tested by: pho (previous version) Notes: svn path=/head/; revision=313269
* mtx: reduce lock accessesMateusz Guzik2017-01-031-1/+7
| | | | | | | | | | | | | | | | | Instead of spuriously re-reading the lock value, read it once. This change also has a side effect of fixing a performance bug: on failed _mtx_obtain_lock, it was possible that re-read would find the lock is unowned, but in this case the primitive would make a trip through turnstile code. This is diff reduction to a variant which uses atomic_fcmpset. Discussed with: jhb (previous version) Tested by: pho (previous version) Notes: svn path=/head/; revision=311172
* Implement mtx_trylock_spin(9).Konstantin Belousov2016-07-231-0/+44
| | | | | | | | | | | Discussed with: bde Reviewed by: jhb Sponsored by: The FreeBSD Foundation MFC after: 1 week Differential revision: https://reviews.freebsd.org/D7192 Notes: svn path=/head/; revision=303211
* Microoptimize locking primitives by avoiding unnecessary atomic ops.Mateusz Guzik2016-06-011-3/+3
| | | | | | | | | | | | | Inline version of primitives do an atomic op and if it fails they fallback to actual primitives, which immediately retry the atomic op. The obvious optimisation is to check if the lock is free and only then proceed to do an atomic op. Reviewed by: jhb, vangyzen Notes: svn path=/head/; revision=301157
* Implement the lockstat provider using SDT(9) instead of the custom providerMark Johnston2015-07-191-13/+10
| | | | | | | | | | | in lockstat.ko. This means that lockstat probes now have typed arguments and will utilize SDT probe hot-patching support when it arrives. Reviewed by: gnn Differential Revision: https://reviews.freebsd.org/D2993 Notes: svn path=/head/; revision=285703
* Add _NEW flag to mtx(9), sx(9), rmlock(9) and rwlock(9).Dmitry Chagin2014-12-131-0/+1
| | | | | | | | | | | A _NEW flag passed to _init_flags() to avoid check for double-init. Differential Revision: https://reviews.freebsd.org/D1208 Reviewed by: jhb, wblock MFC after: 1 Month Notes: svn path=/head/; revision=275751
* Garbage collect mtxpool_lockbuilder, the mutex pool historically usedRobert Watson2014-05-021-4/+0
| | | | | | | | | | | | | for lockmgr and sx interlocks, but unused since optimised versions of those sleep locks were introduced. This will save a (quite) small amount of memory in all kernel configurations. The sleep mutex pool is retained as it is used for 'struct bio' and several other consumers. Discussed with: jhb MFC after: 3 days Notes: svn path=/head/; revision=265216
* fix spelling of lock_initialized.. jhb approved..John-Mark Gurney2014-01-281-1/+1
| | | | | | | MFC after: 1 week Notes: svn path=/head/; revision=261238
* - For kernel compiled only with KDTRACE_HOOKS and not any lock debuggingAttilio Rao2013-11-251-0/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | option, unbreak the lock tracing release semantic by embedding calls to LOCKSTAT_PROFILE_RELEASE_LOCK() direclty in the inlined version of the releasing functions for mutex, rwlock and sxlock. Failing to do so skips the lockstat_probe_func invokation for unlocking. - As part of the LOCKSTAT support is inlined in mutex operation, for kernel compiled without lock debugging options, potentially every consumer must be compiled including opt_kdtrace.h. Fix this by moving KDTRACE_HOOKS into opt_global.h and remove the dependency by opt_kdtrace.h for all files, as now only KDTRACE_FRAMES is linked there and it is only used as a compile-time stub [0]. [0] immediately shows some new bug as DTRACE-derived support for debug in sfxge is broken and it was never really tested. As it was not including correctly opt_kdtrace.h before it was never enabled so it was kept broken for a while. Fix this by using a protection stub, leaving sfxge driver authors the responsibility for fixing it appropriately [1]. Sponsored by: EMC / Isilon storage division Discussed with: rstone [0] Reported by: rstone [1] Discussed with: philip Notes: svn path=/head/; revision=258541
* Remove an unused macro we originally got from BSD/OS.John Baldwin2013-06-011-6/+0
| | | | Notes: svn path=/head/; revision=251220
* MFcalloutng:Davide Italiano2013-03-041-1/+2
| | | | | | | | | | | Introduce sbt variants of msleep(), msleep_spin(), pause(), tsleep() in the KPI, allowing to specify timeout in 'sbintime_t' rather than ticks. Sponsored by: Google Summer of Code 2012, iXsystems inc. Tested by: flo, marius, ian, markj, Fabian Keil Notes: svn path=/head/; revision=247787
* Tweak comment to make more clear why it will fail.Attilio Rao2012-11-021-1/+1
| | | | | | | Submitted by: jimharris Notes: svn path=/head/; revision=242483
* Give mtx(9) the ability to crunch different type of structures, with theAttilio Rao2012-10-311-16/+53
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | only constraint that they have a lock cookie named mtx_lock. This name, then, becames reserved from the struct that wants to use the mtx(9) KPI and other locking primitives cannot reuse it for their members. Namely such structs are the current struct mtx and the new struct mtx_padalign. The new structure will define an object which is the same as the same layout of a struct mtx but will be allocated in areas aligned to the cache line size and will be as big as a cache line. This is supposed to give higher performance for highly contented mutexes both spin or sleep (because of the adaptive spinning), where the cache line contention results in too much traffic on the system bus. The struct mtx_padalign can be used in a completely transparent way with the mtx(9) KPI. At the moment, a possibility to MFC the patch should be carefully evaluated because this patch breaks the low level KPI (not its representation though). Discussed with: jhb Reviewed by: jeff, andre Reviewed by: mdf (earlier version) Tested by: jimharris Notes: svn path=/head/; revision=242395
* panic: add a switch and infrastructure for stopping other CPUs in SMP caseAndriy Gapon2011-12-111-1/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Historical behavior of letting other CPUs merily go on is a default for time being. The new behavior can be switched on via kern.stop_scheduler_on_panic tunable and sysctl. Stopping of the CPUs has (at least) the following benefits: - more of the system state at panic time is preserved intact - threads and interrupts do not interfere with dumping of the system state Only one thread runs uninterrupted after panic if stop_scheduler_on_panic is set. That thread might call code that is also used in normal context and that code might use locks to prevent concurrent execution of certain parts. Those locks might be held by the stopped threads and would never be released. To work around this issue, it was decided that instead of explicit checks for panic context, we would rather put those checks inside the locking primitives. This change has substantial portions written and re-written by attilio and kib at various times. Other changes are heavily based on the ideas and patches submitted by jhb and mdf. bde has provided many insights into the details and history of the current code. The new behavior may cause problems for systems that use a USB keyboard for interfacing with system console. This is because of some unusual locking patterns in the ukbd code which have to be used because on one hand ukbd is below syscons, but on the other hand it has to interface with other usb code that uses regular mutexes/Giant for its concurrency protection. Dumping to USB-connected disks may also be affected. PR: amd64/139614 (at least) In cooperation with: attilio, jhb, kib, mdf Discussed with: arch@, bde Tested by: Eugene Grosbein <eugen@grosbein.net>, gnn, Steven Hartland <killing@multiplay.co.uk>, glebius, Andrew Boyer <aboyer@averesystems.com> (various versions of the patch) MFC after: 3 months (or never) Notes: svn path=/head/; revision=228424
* Introduce macro stubs in the mutex implementation that will be alwaysAttilio Rao2011-11-201-30/+45
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | defined and will allow consumers, willing to provide options, file and line to locking requests, to not worry about options redefining the interfaces. This is typically useful when there is the need to build another locking interface on top of the mutex one. The introduced functions that consumers can use are: - mtx_lock_flags_ - mtx_unlock_flags_ - mtx_lock_spin_flags_ - mtx_unlock_spin_flags_ - mtx_assert_ - thread_lock_flags_ Spare notes: - Likely we can get rid of all the 'INVARIANTS' specification in the ppbus code by using the same macro as done in this patch (but this is left to the ppbus maintainer) - all the other locking interfaces may require a similar cleanup, where the most notable case is sx which will allow a further cleanup of vm_map locking facilities - The patch should be fully compatible with older branches, thus a MFC is previewed (infact it uses all the underlying mechanisms already present). Comments review by: eadler, Ben Kaduk Discussed with: kib, jhb MFC after: 1 month Notes: svn path=/head/; revision=227758
* Constify arguments for locking KPIs where possible.Pawel Jakub Dawidek2011-11-161-1/+1
| | | | | | | | | | This enables locking consumers to pass their own structures around as const and be able to assert locks embedded into those structures. Reviewed by: ed, kib, jhb Notes: svn path=/head/; revision=227588
* Retire mp_fixme(). It's no longer used.Alan Cox2011-02-141-10/+0
| | | | Notes: svn path=/head/; revision=218673
* - Remove <machine/mutex.h>. Most of the headers were empty, and theJohn Baldwin2010-11-091-63/+36
| | | | | | | | | | | | | | | contents of the ones that were not empty were stale and unused. - Now that <machine/mutex.h> no longer exists, there is no need to allow it to override various helper macros in <sys/mutex.h>. - Rename various helper macros for low-level operations on mutexes to live in the _mtx_* or __mtx_* namespaces. While here, change the names to more closely match the real API functions they are backing. - Drop support for including <sys/mutex.h> in assembly source files. Suggested by: bde (1, 2) Notes: svn path=/head/; revision=215054
* Account for unlocking a spin mutex in the lock profiling code in the !SMPJohn Baldwin2010-09-291-1/+4
| | | | | | | | | | case. Submitted by: truckman MFC after: 3 days Notes: svn path=/head/; revision=213271
* In non-debugging mode make this define (void)0 instead of nothing. ThisRoman Divacky2009-06-211-1/+1
| | | | | | | | | | | | | helps to catch bugs like the below with clang. if (cond); <--- note the trailing ; something(); Approved by: ed (mentor) Discussed on: current@ Notes: svn path=/head/; revision=194578
* Add the OpenSolaris dtrace lockstat provider. The lockstat providerStacey Son2009-05-261-7/+9
| | | | | | | | | | | | | adds probes for mutexes, reader/writer and shared/exclusive locks to gather contention statistics and other locking information for dtrace scripts, the lockstat(1M) command and other potential consumers. Reviewed by: attilio jhb jb Approved by: gnn (mentor) Notes: svn path=/head/; revision=192853
* - Embed the recursion counter for any locking primitive directly in theAttilio Rao2008-05-151-0/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | lock_object, using an unified field called lo_data. - Replace lo_type usage with the w_name usage and at init time pass the lock "type" directly to witness_init() from the parent lock init function. Handle delayed initialization before than witness_initialize() is called through the witness_pendhelp structure. - Axe out LO_ENROLLPEND as it is not really needed. The case where the mutex init delayed wants to be destroyed can't happen because witness_destroy() checks for witness_cold and panic in case. - In enroll(), if we cannot allocate a new object from the freelist, notify that to userspace through a printf(). - Modify the depart function in order to return nothing as in the current CVS version it always returns true and adjust callers accordingly. - Fix the witness_addgraph() argument name prototype. - Remove unuseful code from itismychild(). This commit leads to a shrinked struct lock_object and so smaller locks, in particular on amd64 where 2 uintptr_t (16 bytes per-primitive) are gained. Reviewed by: jhb Notes: svn path=/head/; revision=179025
* Unify assertion flags for all the main primitives using the LA_* underlyingAttilio Rao2007-11-191-4/+4
| | | | | | | | family of macros. This will allow to use unified flags for assertions with the generic locking primitive class. Notes: svn path=/head/; revision=173759
* Remove the now-unused NET_{LOCK,UNLOCK,ASSERT}_GIANT() macros, whichRobert Watson2007-08-061-13/+0
| | | | | | | | | | | | | | | | | | previously conditionally acquired Giant based on debug.mpsafenet. As that has now been removed, they are no longer required. Removing them significantly simplifies error-handling in the socket layer, eliminated quite a bit of unwinding of locking in error cases. While here clean up the now unneeded opt_net.h, which previously was used for the NET_WITH_GIANT kernel option. Clean up some related gotos for consistency. Reviewed by: bz, csjp Tested by: kris Approved by: re (kensmith) Notes: svn path=/head/; revision=171744
* Replace references to NET_CALLOUT_MPSAFE with CALLOUT_MPSAFE, and removeRobert Watson2007-07-281-1/+0
| | | | | | | | | | | definition of NET_CALLOUT_MPSAFE, which is no longer required now that debug.mpsafenet has been removed. The once over: bz Approved by: re (kensmith) Notes: svn path=/head/; revision=171637
* First in a series of changes to remove the now-unused Giant compatibilityRobert Watson2007-07-271-22/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | framework for non-MPSAFE network protocols: - Remove debug_mpsafenet variable, sysctl, and tunable. - Remove NET_NEEDS_GIANT() and associate SYSINITSs used by it to force debug.mpsafenet=0 if non-MPSAFE protocols are compiled into the kernel. - Remove logic to automatically flag interrupt handlers as non-MPSAFE if debug.mpsafenet is set for an INTR_TYPE_NET handler. - Remove logic to automatically flag netisr handlers as non-MPSAFE if debug.mpsafenet is set. - Remove references in a few subsystems, including NFS and Cronyx drivers, which keyed off debug_mpsafenet to determine various aspects of their own locking behavior. - Convert NET_LOCK_GIANT(), NET_UNLOCK_GIANT(), and NET_ASSERT_GIANT into no-op's, as their entire behavior was determined by the value in debug_mpsafenet. - Alias NET_CALLOUT_MPSAFE to CALLOUT_MPSAFE. Many remaining references to NET_.*_GIANT() and NET_CALLOUT_MPSAFE are still present in subsystems, and will be removed in followup commits. Reviewed by: bz, jhb Approved by: re (kensmith) Notes: svn path=/head/; revision=171613
* - Remove the global definition of sched_lock in mutex.h to breakJeff Roberson2007-07-181-1/+0
| | | | | | | | | | | | | | new code and third party modules which try to depend on it. - Initialize sched_lock in sched_4bsd.c. - Declare sched_lock in sparc64 pmap.c and assert that we're compiling with SCHED_4BSD to prevent accidental crashes from running ULE. This is the sole remaining file outside of the scheduler that uses the global sched_lock. Approved by: re Notes: svn path=/head/; revision=171488
* - Placing the 'volatile' on the right side of the * in the td_lockJeff Roberson2007-06-061-1/+1
| | | | | | | | | declaration removes the need for __DEVOLATILE(). Pointed out by: tegge Notes: svn path=/head/; revision=170358
* Commit 1/14 of sched_lock decomposition.Jeff Roberson2007-06-041-0/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | - Move all scheduler locking into the schedulers utilizing a technique similar to solaris's container locking. - A per-process spinlock is now used to protect the queue of threads, thread count, suspension count, p_sflags, and other process related scheduling fields. - The new thread lock is actually a pointer to a spinlock for the container that the thread is currently owned by. The container may be a turnstile, sleepqueue, or run queue. - thread_lock() is now used to protect access to thread related scheduling fields. thread_unlock() unlocks the lock and thread_set_lock() implements the transition from one lock to another. - A new "blocked_lock" is used in cases where it is not safe to hold the actual thread's lock yet we must prevent access to the thread. - sched_throw() and sched_fork_exit() are introduced to allow the schedulers to fix-up locking at these points. - Add some minor infrastructure for optionally exporting scheduler statistics that were invaluable in solving performance problems with this patch. Generally these statistics allow you to differentiate between different causes of context switches. Tested by: kris, current@ Tested on: i386, amd64, ULE, 4BSD, libthr, libkse, PREEMPTION, etc. Discussed with: kris, attilio, kmacy, jhb, julian, bde (small parts each) Notes: svn path=/head/; revision=170293
* Group the loop to acquire/release Giant with the WITNESS_SAVE/RESTORE underJohn Baldwin2007-04-111-8/+10
| | | | | | | | | | | | | a single conditional. The two operations are linked, but since the link is not very direct, Coverity can't see it. Humans might also miss the link as well. So, this isn't fixing any actual bugs, just improving readability. CID: 1787 (likely others as well) Found by: Coverity Prevent (tm) Notes: svn path=/head/; revision=168619
* move lock_profile calls out of the macros and into kern_mutex.cKip Macy2007-04-031-16/+10
| | | | | | | add check for mtx_recurse == 0 when releasing sleep lock Notes: svn path=/head/; revision=168329
* - Use PARTIAL_PICKUP_GIANT() to implement PICKUP_GIANT().John Baldwin2007-03-301-13/+9
| | | | | | | | - Move UGAR() macro up to the comment that describes it. - Fix a couple of typos. Notes: svn path=/head/; revision=168074
* Rename the 'mtx_object', 'rw_object', and 'sx_object' members of mutexes,John Baldwin2007-03-211-10/+10
| | | | | | | rwlocks, and sx locks to 'lock_object'. Notes: svn path=/head/; revision=167787
generated by cgit v1.2.3 (git 2.25.1) at 2024-05-17 11:28:59 +0000