path: root/sys/sys/lock.h
Commit message (Collapse)AuthorAgeFilesLines
* sys: clean up empty lines in .c and .h filesMateusz Guzik2020-09-011-1/+0
| | | | Notes: svn path=/head/; revision=365223
* locks: fix a long standing bug for primitives with kdtrace but without spinningMateusz Guzik2020-07-231-0/+7
| | | | | | | | | | | | | In such a case the second argument to lock_delay_arg_init was NULL which was immediately causing a null pointer deref. Since the sructure is only used for spin count, provide a dedicate routine initializing it. Reported by: andrew Notes: svn path=/head/; revision=363451
* Move MPASS() macros to systm.h. They are widely used all overGleb Smirnoff2020-06-081-10/+0
| | | | | | | | | | the kernel and aren't contained only to the locking code. Reviewed by: kib, mjg Differential Revision: https://reviews.freebsd.org/D23656 Notes: svn path=/head/; revision=361931
* locks: add default delay structMateusz Guzik2020-01-051-0/+4
| | | | | | | Use it for all primitives. This makes everything fit in 8 bytes. Notes: svn path=/head/; revision=356375
* locks: convert delay times to u_shortMateusz Guzik2020-01-051-3/+3
| | | | | | | int is just a waste of space for this purpose. Notes: svn path=/head/; revision=356374
* witness: sleepable rm locks are not sleepable in read modeRyan Libby2019-11-271-0/+1
| | | | | | | | | | | | | | | | | | | | There are two classes of rm lock, one "sleepable" and one not. But even a "sleepable" rm lock is only sleepable in write mode, and is non-sleepable when taken in read mode. Warn about sleepable rm locks in read mode as non-sleepable locks. Do this by defining a new lock operation flag, LOP_NOSLEEP, to indicate that a lock is non-sleepable despite what the LO_SLEEPABLE flag would indicate, and defining a new witness lock instance flag, LI_SLEEPABLE, to track the product of LO_SLEEPABLE and LOP_NOSLEEP on the lock instance. Reviewed by: markj Sponsored by: Dell EMC Isilon Differential Revision: https://reviews.freebsd.org/D22527 Notes: svn path=/head/; revision=355126
* Extract eventfilter declarations to sys/_eventfilter.hConrad Meyer2019-05-201-32/+0
| | | | | | | | | | | | | | | | | | | | | | | | This allows replacing "sys/eventfilter.h" includes with "sys/_eventfilter.h" in other header files (e.g., sys/{bus,conf,cpu}.h) and reduces header pollution substantially. EVENTHANDLER_DECLARE and EVENTHANDLER_LIST_DECLAREs were moved out of .c files into appropriate headers (e.g., sys/proc.h, powernv/opal.h). As a side effect of reduced header pollution, many .c files and headers no longer contain needed definitions. The remainder of the patch addresses adding appropriate includes to fix those files. LOCK_DEBUG and LOCK_FILE_LINE_ARG are moved to sys/_lock.h, as required by sys/mutex.h since r326106 (but silently protected by header pollution prior to this change). No functional change (intended). Of course, any out of tree modules that relied on header pollution for sys/eventhandler.h, sys/lock.h, or sys/mutex.h inclusion need to be fixed. __FreeBSD_version has been bumped. Notes: svn path=/head/; revision=347984
* inline atomics and allow tied modules to inline locksMatt Macy2018-07-021-1/+1
| | | | | | | | | | | | | | - inline atomics in modules on i386 and amd64 (they were always inline on other arches) - allow modules to opt in to inlining locks by specifying MODULE_TIED=1 in the makefile Reviewed by: kib Sponsored by: Limelight Networks Differential Revision: https://reviews.freebsd.org/D16079 Notes: svn path=/head/; revision=335873
* Start witness much earlier in boot so that we can shrink the pend list andJeff Roberson2018-03-221-0/+2
| | | | | | | | | | make it more immune to further change. Reviewed by: markj, imp (Part of D14707) Sponsored by: Netflix, Dell/EMC Isilon Notes: svn path=/head/; revision=331368
* sys/sys: further adoption of SPDX licensing ID tags.Pedro F. Giffuni2017-11-271-0/+2
| | | | | | | | | | | | | | | Mainly focus on files that use BSD 2-Clause license, however the tool I was using misidentified many licenses so this was mostly a manual - error prone - task. The Software Package Data Exchange (SPDX) group provides a specification to make it easier for automated tools to detect and summarize well known opensource licenses. We are gradually adopting the specification, noting that the tags are considered only advisory and do not, in any way, superceed or replace the license texts. Notes: svn path=/head/; revision=326256
* locks: remove the file + line argument from internal primitives when not usedMateusz Guzik2017-11-221-0/+4
| | | | | | | | | | | | | The pair is of use only in debug or LOCKPROF kernels, but was passed (zeroed) for many locks even in production kernels. While here whack the tid argument from wlock hard and xlock hard. There is no kbi change of any sort - "external" primitives still accept the pair. Notes: svn path=/head/; revision=326106
* locks: take the number of readers into account when waitingMateusz Guzik2017-10-051-0/+7
| | | | | | | | | | | | | | | | Previous code would always spin once before checking the lock. But a lock with e.g. 6 readers is not going to become free in the duration of once spin even if they start draining immediately. Conservatively perform one for each reader. Note that the total number of allowed spins is still extremely small and is subject to change later. MFC after: 1 week Notes: svn path=/head/; revision=324335
* Do not turn LOCK_DEBUG for INVARIANT_SUPPORT. This allows to addGleb Smirnoff2017-03-091-1/+1
| | | | | | | | | INVARIANT_SUPPORT into a kernel without performance penalties. Discussed with: jhb, mjg Notes: svn path=/head/; revision=314932
* locks: fix compilation with KTR wihout KTR_LOCKSMateusz Guzik2017-03-011-1/+2
| | | | | | | | | While here wrap the overly long line. Reported by: np Notes: svn path=/head/; revision=314476
* mtx: plug the 'opts' argument when not usedMateusz Guzik2017-02-181-0/+5
| | | | Notes: svn path=/head/; revision=313908
* locks: change backoff to exponentialMateusz Guzik2017-02-071-5/+9
| | | | | | | | | | | | | | | Previous implementation would use a random factor to spread readers and reduce chances of starvation. This visibly reduces effectiveness of the mechanism. Switch to the more traditional exponential variant. Try to limit starvation by imposing an upper limit of spins after which spinning is half of what other threads get. Note the mechanism is turned off by default. Reviewed by: kib (previous version) Notes: svn path=/head/; revision=313386
* Implement trivial backoff for locking primitives.Mateusz Guzik2016-08-011-0/+24
| | | | | | | | | | | | | | | | | | | | | | | | All current spinning loops retry an atomic op the first chance they get, which leads to performance degradation under load. One classic solution to the problem consists of delaying the test to an extent. This implementation has a trivial linear increment and a random factor for each attempt. For simplicity, this first thouch implementation only modifies spinning loops where the lock owner is running. spin mutexes and thread lock were not modified. Current parameters are autotuned on boot based on mp_cpus. Autotune factors are very conservative and are subject to change later. Reviewed by: kib, jhb Tested by: pho MFC after: 1 week Notes: svn path=/head/; revision=303643
* Add _NEW flag to mtx(9), sx(9), rmlock(9) and rwlock(9).Dmitry Chagin2014-12-131-0/+1
| | | | | | | | | | | A _NEW flag passed to _init_flags() to avoid check for double-init. Differential Revision: https://reviews.freebsd.org/D1208 Reviewed by: jhb, wblock MFC after: 1 Month Notes: svn path=/head/; revision=275751
* fix spelling of lock_initialized.. jhb approved..John-Mark Gurney2014-01-281-1/+1
| | | | | | | MFC after: 1 week Notes: svn path=/head/; revision=261238
* Right now LOCK_DEBUG is enabled if KTR is on.Attilio Rao2013-12-021-1/+2
| | | | | | | | | | | | | | | This is to support LOCK_LOG_* functionality effectively in debugging environments but it is overkill because really LOCK_DEBUG should be on only if (KTR_COMPILE & KTR_LOCK) is true. Fix this by applying the correct logic. In this process, move the KTR classes to its own header to reduce namespace pollution. Sponsored by: EMC / Isilon storage division Reviewed by: jhb Notes: svn path=/head/; revision=258858
* Fix lc_lock/lc_unlock() support for rmlocks held in shared mode. WithDavide Italiano2013-09-201-7/+8
| | | | | | | | | | | | | | | | | | current lock classes KPI it was really difficult because there was no way to pass an rmtracker object to the lock/unlock routines. In order to accomplish the task, modify the aforementioned functions so that they can return (or pass as argument) an uinptr_t, which is in the rm case used to hold a pointer to struct rm_priotracker for current thread. As an added bonus, this fixes rm_sleep() in the rm shared case, which right now can communicate priotracker structure between lc_unlock()/lc_lock(). Suggested by: jhb Reviewed by: jhb Approved by: re (delphij) Notes: svn path=/head/; revision=255745
* Several improvements to rmlock(9). Many of these are based on patchesJohn Baldwin2013-06-251-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | provided by Isilon. - Add an rm_assert() supporting various lock assertions similar to other locking primitives. Because rmlocks track readers the assertions are always fully accurate unlike rw_assert() and sx_assert(). - Flesh out the lock class methods for rmlocks to support sleeping via condvars and rm_sleep() (but only while holding write locks), rmlock details in 'show lock' in DDB, and the lc_owner method used by dtrace. - Add an internal destroyed cookie so that API functions can assert that an rmlock is not destroyed. - Make use of rm_assert() to add various assertions to the API (e.g. to assert locks are held when an unlock routine is called). - Give RM_SLEEPABLE locks their own lock class and always use the rmlock's own lock_object with WITNESS. - Use THREAD_NO_SLEEPING() / THREAD_SLEEPING_OK() to disallow sleeping while holding a read lock on an rmlock. Submitted by: andre Obtained from: EMC/Isilon Notes: svn path=/head/; revision=252209
* Add option WITNESS_NO_VNODE to suppress printing LORs between VNODEMarcel Moolenaar2013-05-091-0/+1
| | | | | | | | | | | | locks. To support this, VNODE locks are created with the LK_IS_VNODE flag. This flag is propagated down using the LO_IS_VNODE flag. Note that WITNESS still records the LOR. Only the printing and the optional entering into the kernel debugger is bypassed with the WITNESS_NO_VNODE option. Notes: svn path=/head/; revision=250411
* Remove unused/obsolete macros.Attilio Rao2012-11-131-11/+0
| | | | | | | MFC after: 3 days Notes: svn path=/head/; revision=242988
* Constify arguments for locking KPIs where possible.Pawel Jakub Dawidek2011-11-161-4/+4
| | | | | | | | | | This enables locking consumers to pass their own structures around as const and be able to assert locks embedded into those structures. Reviewed by: ed, kib, jhb Notes: svn path=/head/; revision=227588
* Protect BSDI $Id from accidental manipulation.David E. O'Brien2010-09-101-1/+1
| | | | Notes: svn path=/head/; revision=212408
* Right now, WITNESS just blindly pipes all the output to theAttilio Rao2010-05-111-2/+4
| | | | | | | | | | | | | | | | (TOCONS | TOLOG) mask even when called from DDB points. That breaks several output, where the most notable is textdump output. Fix this by having configurable callbacks passed to witness_list_locks() and witness_display_spinlock() for printing out datas. Reported by: several broken textdump outputs Tested by: Giovanni Trematerra <giovanni dot trematerra at gmail dot com> MFC after: 7 days X-MFC: r207922 Notes: svn path=/head/; revision=207929
* Fix style for witness functions prototypes.Attilio Rao2010-05-111-2/+2
| | | | | | | MFC after: 7 days Notes: svn path=/head/; revision=207925
* Add the address of the lock to the KTR_LOCK trace.Konstantin Belousov2009-08-141-4/+4
| | | | | | | | Tested by: pho Approved by: re (rwatson) Notes: svn path=/head/; revision=196204
* In non-debugging mode make this define (void)0 instead of nothing. ThisRoman Divacky2009-06-211-13/+13
| | | | | | | | | | | | | helps to catch bugs like the below with clang. if (cond); <--- note the trailing ; something(); Approved by: ed (mentor) Discussed on: current@ Notes: svn path=/head/; revision=194578
* Add the OpenSolaris dtrace lockstat provider. The lockstat providerStacey Son2009-05-261-0/+1
| | | | | | | | | | | | | adds probes for mutexes, reader/writer and shared/exclusive locks to gather contention statistics and other locking information for dtrace scripts, the lockstat(1M) command and other potential consumers. Reviewed by: attilio jhb jb Approved by: gnn (mentor) Notes: svn path=/head/; revision=192853
* Add functions WITNESS so it can be asserted that the lock is not released for aAndrew Thompson2009-01-211-0/+10
| | | | | | | | | | | | | | section of code, this uses WITNESS_NORELEASE() and WITNESS_RELEASEOK() to mark the boundaries. Both functions require the lock to be held when calling. This is intended for scenarios like a bus asserting that the bus lock is not dropped during a driver call. There doesn't appear to be a man page to document this in. Reviewed by: jhb Notes: svn path=/head/; revision=187511
* Teach WITNESS about the interlocks used with lockmgr. This removes a bunchJohn Baldwin2008-09-101-6/+7
| | | | | | | | | | | of spurious witness warnings since lockmgr grew witness support. Before this, every time you passed an interlock to a lockmgr lock WITNESS treated it as a LOR. Reviewed by: attilio Notes: svn path=/head/; revision=182914
* Introduce some WITNESS improvements:Attilio Rao2008-08-131-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | - Speedup the lock orderings lookup modifying the witness graph from a linked tree to a matrix. A table lookup caches the lock orderings in order to make a O(1) access for them. Any witness object has an unique index withing this lookup cache table. - Reduce the lock contention on w_mtx acquiring it only when the LOR actually happens and not in a sane case. In order to do this don't totally flush lock lists (per-CPU spinlocks list and per-thread sleeplocks list) but check for ll_count anytime we need to have to verify allocations sanity. - Introduce the function witness_thread_exit() in the witness namespace which should verify a thread doesn't hold any witness occurrence why exiting. - Rename the sysctl debug.witness.graphs into debug.witness.fullgraph and add debug.witness.badstacks which prints out stacks for LOR revealed. This is implemented using the stack(9) support, which makes WITNESS to be dependent by the STACK option or by the DDB (including STACK) option. - Fix style(9) for src/sys/kern/subr_witness.c The hash table approach has been developed by Ilya Maykov on the behalf of Isilon Systems which kindly released the patch. Jeff Roberson, ported the patch to -CURRENT and fixed w_mtx contention, on the behalf of Nokia. Submitted by: Ilya Maykov <ivmaykov at gmail dot com> (Isilon Systems), jeff Sponsored by: Nokia Notes: svn path=/head/; revision=181695
* - Embed the recursion counter for any locking primitive directly in theAttilio Rao2008-05-151-5/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | lock_object, using an unified field called lo_data. - Replace lo_type usage with the w_name usage and at init time pass the lock "type" directly to witness_init() from the parent lock init function. Handle delayed initialization before than witness_initialize() is called through the witness_pendhelp structure. - Axe out LO_ENROLLPEND as it is not really needed. The case where the mutex init delayed wants to be destroyed can't happen because witness_destroy() checks for witness_cold and panic in case. - In enroll(), if we cannot allocate a new object from the freelist, notify that to userspace through a printf(). - Modify the depart function in order to return nothing as in the current CVS version it always returns true and adjust callers accordingly. - Fix the witness_addgraph() argument name prototype. - Remove unuseful code from itismychild(). This commit leads to a shrinked struct lock_object and so smaller locks, in particular on amd64 where 2 uintptr_t (16 bytes per-primitive) are gained. Reviewed by: jhb Notes: svn path=/head/; revision=179025
* struct lock_instance and struct lock_list_entry don't need to be in theAttilio Rao2008-04-131-34/+1
| | | | | | | | | public namespace for WITNESS as they are only used internally so just move them in the private namespace for the subsystem (with all related supporting definitions). Notes: svn path=/head/; revision=178165
* - Add real assertions to lockmgr locking primitives.Attilio Rao2008-02-131-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | A couple of notes for this: * WITNESS support, when enabled, is only used for shared locks in order to avoid problems with the "disowned" locks * KA_HELD and KA_UNHELD only exists in the lockmgr namespace in order to assert for a generic thread (not curthread) owning or not the lock. Really, this kind of check is bogus but it seems very widespread in the consumers code. So, for the moment, we cater this untrusted behaviour, until the consumers are not fixed and the options could be removed (hopefully during 8.0-CURRENT lifecycle) * Implementing KA_HELD and KA_UNHELD (not surported natively by WITNESS) made necessary the introduction of LA_MASKASSERT which specifies the range for default lock assertion flags * About other aspects, lockmgr_assert() follows exactly what other locking primitives offer about this operation. - Build real assertions for buffer cache locks on the top of lockmgr_assert(). They can be used with the BUF_ASSERT_*(bp) paradigm. - Add checks at lock destruction time and use a cookie for verifying lock integrity at any operation. - Redefine BUF_LOCKFREE() in order to not use a direct assert but let it rely on the aforementioned destruction time check. KPI results evidently broken, so __FreeBSD_version bumping and manpage update result necessary and will be committed soon. Side note: lockmgr_assert() will be used soon in order to implement real assertions in the vnode namespace replacing the legacy and still bogus "VOP_ISLOCKED()" way. Tested by: kris (earlier version) Reviewed by: jhb Notes: svn path=/head/; revision=176249
* Expand lock class with the "virtual" function lc_assert which will offerAttilio Rao2007-11-181-0/+1
| | | | | | | | | | | | an unified way for all the lock primitives to express lock assertions. Currenty, lockmgrs and rmlocks don't have assertions, so just panic in that case. This will be a base for more callout improvements. Ok'ed by: jhb, jeff Notes: svn path=/head/; revision=173733
* Initial checkin for rmlock (read mostly lock) a multi reader single writerStephan Uphoff2007-11-081-0/+1
| | | | | | | | | | | lock optimized for almost exclusive reader access. (see also rmlock.9) TODO: Convert to per cpu variables linkerset as soon as it is available. Optimize UP (single processor) case. Notes: svn path=/head/; revision=173444
* Remove unneccessary LO_CONTESTED flagKip Macy2007-04-031-1/+0
| | | | Notes: svn path=/head/; revision=168310
* Make use of 'lock_object' being the same field name in the witness_check*()John Baldwin2007-03-211-8/+4
| | | | | | | | | | | | macros. - witness_check() replaces witness_check_mtx() and witness_check_exclusive_sx() and checks for an exclusive acquire of either a mutex, rwlock, or sx lock. - witness_check_shared() replaces witness_check_shared_sx() and checks for a shared acquire of either a rwlock or sx lock. Notes: svn path=/head/; revision=167788
* Rename the 'mtx_object', 'rw_object', and 'sx_object' members of mutexes,John Baldwin2007-03-211-3/+3
| | | | | | | rwlocks, and sx locks to 'lock_object'. Notes: svn path=/head/; revision=167787
* Add two new function pointers 'lc_lock' and 'lc_unlock' to lock classes.John Baldwin2007-03-091-0/+9
| | | | | | | | | | | | | | | | These functions are intended to be used to drop a lock and then reacquire it when doing an sleep such as msleep(9). Both functions accept a 'struct lock_object *' as their first parameter. The 'lc_unlock' function returns an integer that is then passed as the second paramter to the subsequent 'lc_lock' function. This can be used to communicate state. For example, sx locks and rwlocks use this to indicate if the lock was share/read locked vs exclusive/write locked. Currently, spin mutexes and lockmgr locks do not provide working lc_lock and lc_unlock functions. Notes: svn path=/head/; revision=167368
* Further improvements to LOCK_PROFILING:Kip Macy2007-02-271-0/+1
| | | | | | | | | | | | | | | - Fix missing initialization in kern_rwlock.c causing bogus times to be collected - Move updates to the lock hash to after the lock is released for spin mutexes, sleep mutexes, and sx locks - Add new kernel build option LOCK_PROFILE_FAST - only update lock profiling statistics when an acquisition is contended. This reduces the overhead of LOCK_PROFILING to increasing system time by 20%-25% which on "make -j8 kernel-toolchain" on a dual woodcrest is unmeasurable in terms of wall-clock time. Contrast this to enabling lock profiling without LOCK_PROFILE_FAST and I see a 5x-6x slowdown in wall-clock time. Notes: svn path=/head/; revision=167054
* Style; remove argument names from prototype, be consistent withBruce M Simpson2007-02-031-3/+3
| | | | | | | | | | | | rest of file. This has the additional side-effect of removing a C++ reserved keyword from this file, which prevents the Click Modular Router's FreeBSD kernel support from building. Reviewed by: silence on -current Notes: svn path=/head/; revision=166455
* track lock class name in a way that doesn't break WITNESSKip Macy2006-11-131-0/+1
| | | | Notes: svn path=/head/; revision=164246
* MUTEX_PROFILING has been generalized to LOCK_PROFILING. We now profileKip Macy2006-11-111-1/+2
| | | | | | | | | | | | | | wait (time waited to acquire) and hold times for *all* kernel locks. If the architecture has a system synchronized TSC, the profiling code will use that - thereby minimizing profiling overhead. Large chunks of profiling code have been moved out of line, the overhead measured on the T1 for when it is compiled in but not enabled is < 1%. Approved by: scottl (standing in for mentor rwatson) Reviewed by: des and jhb Notes: svn path=/head/; revision=164159
* Update comments to indicate that locks are held by threads, not processes.John Baldwin2006-04-171-3/+3
| | | | Notes: svn path=/head/; revision=157824
* Add a basic reader/writer lock implementation to the kernel. ThisJohn Baldwin2006-01-271-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | implementation is by no means perfect as far as some of the algorithms that it uses and the fact that it is missing some functionality (try locks and upgrades/downgrades are not there yet), however it does seem to work in my local testing. There is more detail in the comments in the code, but the short version follows. A reader/writer lock is very much like a regular mutex: it cannot be held across a voluntary sleep; it can be acquired in an interrupt thread; if the lock is held by a writer then the priority of any threads that block on the lock will be lent to the owner; the simple case lock operations all are done in a single atomic op. It also shares some similiarities with sx locks: it supports reader/writer semantics (multiple readers, but single writers); readers are allowed to recurse, but writers are not. We can extend this implementation further by either improving algorithms or adding new functionality, but this should at least give us a base to work with now. Reviewed by: arch (in theory) Tested on: i386 (4 cpu box with a kernel module that used 4 threads that randomly chose between read locks and write locks that ran w/o panicing for over a day solid. It usually panic'd within a few seconds when there were bugs during testing. :) The kernel module source is available on request.) Notes: svn path=/head/; revision=154941
* Add a new macro wrapper WITNESS_CHECK() around the witness_warn() function.John Baldwin2006-01-271-0/+4
| | | | | | | | | | | | | The difference between WITNESS_CHECK() and WITNESS_WARN() is that WITNESS_CHECK() should be used in the places that the return value of witness_warn() is checked, whereas WITNESS_WARN() should be used in places where the return value is ignored. Specifically, in a kernel without WITNESS enabled, WITNESS_WARN() evaluates to an empty string where as WITNESS_CHECK evaluates to 0. I also updated the one place that was checking the return value of WITNESS_WARN() to use WITNESS_CHECK. Notes: svn path=/head/; revision=154934