aboutsummaryrefslogtreecommitdiff
path: root/sys/sys/rwlock.h
Commit message (Collapse)AuthorAgeFilesLines
* rwlock: tidy up r313392Mateusz Guzik2017-02-151-1/+1
| | | | | | | | | | | | While a new bit was added and thread alignment got shifted to accomodate it, RW_READERS_SHIFT was not modified accordingly and clashed with the new flag. This was surprisingly harmless. If the lock was taken for writing, other flags were tested. If the lock was taken for reading, it would correctly work for readers > 1 and this was the only relevant test performed. Notes: svn path=/head/; revision=313784
* rwlock: implement RW_LOCK_WRITER_RECURSED bitMateusz Guzik2017-02-071-8/+5
| | | | | | | | | | This moves recursion handling out of the inlined wunlock path and in particular saves a read and a branch. Discussed with: Notes: svn path=/head/; revision=313392
* rwlock: move lockstat handling out of inline primitivesMateusz Guzik2017-02-051-12/+7
| | | | | | | | | | | | | See r313275 for details. One difference here is that recursion handling was removed from the fallback routine. As it is it was never supposed to see a recursed lock in the first place. Future changes will move it out of inline variants, but right now there is no easy to way to test if the lock is recursed without reading additional words. Notes: svn path=/head/; revision=313282
* rwlock: switch to fcmpsetMateusz Guzik2017-02-051-7/+13
| | | | | | | | Discussed with: jhb Tested by: pho Notes: svn path=/head/; revision=313270
* rwlock: reduce lock accesses similarly to r311172Mateusz Guzik2017-01-181-0/+2
| | | | | | | | Discussed with: jhb Tested by: pho (previous version) Notes: svn path=/head/; revision=312389
* Microoptimize locking primitives by avoiding unnecessary atomic ops.Mateusz Guzik2016-06-011-2/+2
| | | | | | | | | | | | | Inline version of primitives do an atomic op and if it fails they fallback to actual primitives, which immediately retry the atomic op. The obvious optimisation is to check if the lock is free and only then proceed to do an atomic op. Reviewed by: jhb, vangyzen Notes: svn path=/head/; revision=301157
* Consistently use a reader/writer flag for lockstat probes in rwlock(9) andMark Johnston2015-07-191-3/+4
| | | | | | | | sx(9), rather than using the probe function name to determine whether a given lock is a read lock or a write lock. Update lockstat(1) accordingly. Notes: svn path=/head/; revision=285704
* Implement the lockstat provider using SDT(9) instead of the custom providerMark Johnston2015-07-191-5/+4
| | | | | | | | | | | in lockstat.ko. This means that lockstat probes now have typed arguments and will utilize SDT probe hot-patching support when it arrives. Reviewed by: gnn Differential Revision: https://reviews.freebsd.org/D2993 Notes: svn path=/head/; revision=285703
* Add _NEW flag to mtx(9), sx(9), rmlock(9) and rwlock(9).Dmitry Chagin2014-12-131-0/+1
| | | | | | | | | | | A _NEW flag passed to _init_flags() to avoid check for double-init. Differential Revision: https://reviews.freebsd.org/D1208 Reviewed by: jhb, wblock MFC after: 1 Month Notes: svn path=/head/; revision=275751
* Drop the 3rd clause from all 3 clause BSD licenses where I am the soleJohn Baldwin2014-02-051-3/+0
| | | | | | | | | holder to convert them to 2 clause BSD licenses. MFC after: 1 week Notes: svn path=/head/; revision=261520
* fix spelling of lock_initialized.. jhb approved..John-Mark Gurney2014-01-281-1/+1
| | | | | | | MFC after: 1 week Notes: svn path=/head/; revision=261238
* - For kernel compiled only with KDTRACE_HOOKS and not any lock debuggingAttilio Rao2013-11-251-2/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | option, unbreak the lock tracing release semantic by embedding calls to LOCKSTAT_PROFILE_RELEASE_LOCK() direclty in the inlined version of the releasing functions for mutex, rwlock and sxlock. Failing to do so skips the lockstat_probe_func invokation for unlocking. - As part of the LOCKSTAT support is inlined in mutex operation, for kernel compiled without lock debugging options, potentially every consumer must be compiled including opt_kdtrace.h. Fix this by moving KDTRACE_HOOKS into opt_global.h and remove the dependency by opt_kdtrace.h for all files, as now only KDTRACE_FRAMES is linked there and it is only used as a compile-time stub [0]. [0] immediately shows some new bug as DTRACE-derived support for debug in sfxge is broken and it was never really tested. As it was not including correctly opt_kdtrace.h before it was never enabled so it was kept broken for a while. Fix this by using a protection stub, leaving sfxge driver authors the responsibility for fixing it appropriately [1]. Sponsored by: EMC / Isilon storage division Discussed with: rstone [0] Reported by: rstone [1] Discussed with: philip Notes: svn path=/head/; revision=258541
* MFcalloutng:Davide Italiano2013-03-041-1/+2
| | | | | | | | | | | Introduce sbt variants of msleep(), msleep_spin(), pause(), tsleep() in the KPI, allowing to specify timeout in 'sbintime_t' rather than ticks. Sponsored by: Google Summer of Code 2012, iXsystems inc. Tested by: flo, marius, ian, markj, Fabian Keil Notes: svn path=/head/; revision=247787
* Merge r242395,242483 from mutex implementation:Attilio Rao2012-11-031-20/+58
| | | | | | | | | | | | | | | | | | | | give rwlock(9) the ability to crunch different type of structures, with the only constraint that they have a lock cookie named rw_lock. This name, then, becames reserved from the struct that wants to use the rwlock(9) KPI and other locking primitives cannot reuse it for their members. Namely such structs are the current struct rwlock and the new struct rwlock_padalign. The new structure will define an object which has the same layout of a struct rwlock but will be allocated in areas aligned to the cache line size and will be as big as a cache line. For further details check comments on above mentioned revisions. Reviewed by: jimharris, jeff Notes: svn path=/head/; revision=242515
* Constify arguments for locking KPIs where possible.Pawel Jakub Dawidek2011-11-161-2/+2
| | | | | | | | | | This enables locking consumers to pass their own structures around as const and be able to assert locks embedded into those structures. Reviewed by: ed, kib, jhb Notes: svn path=/head/; revision=227588
* When releasing a read/shared lock we need to use a write memory barrierAttilio Rao2009-09-301-7/+0
| | | | | | | | | | | | | in order to avoid, on architectures which doesn't have strong ordered writes, CPU instructions reordering. Diagnosed by: fabio Reviewed by: jhb Tested by: Giovanni Trematerra <giovanni dot trematerra at gmail dot com> Notes: svn path=/head/; revision=197643
* Add the OpenSolaris dtrace lockstat provider. The lockstat providerStacey Son2009-05-261-3/+4
| | | | | | | | | | | | | adds probes for mutexes, reader/writer and shared/exclusive locks to gather contention statistics and other locking information for dtrace scripts, the lockstat(1M) command and other potential consumers. Reviewed by: attilio jhb jb Approved by: gnn (mentor) Notes: svn path=/head/; revision=192853
* add RW_SYSINIT_FLAGS macro and rw_sysinit_flags initialization functionKip Macy2008-12-081-0/+19
| | | | Notes: svn path=/head/; revision=185778
* - Embed the recursion counter for any locking primitive directly in theAttilio Rao2008-05-151-0/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | lock_object, using an unified field called lo_data. - Replace lo_type usage with the w_name usage and at init time pass the lock "type" directly to witness_init() from the parent lock init function. Handle delayed initialization before than witness_initialize() is called through the witness_pendhelp structure. - Axe out LO_ENROLLPEND as it is not really needed. The case where the mutex init delayed wants to be destroyed can't happen because witness_destroy() checks for witness_cold and panic in case. - In enroll(), if we cannot allocate a new object from the freelist, notify that to userspace through a printf(). - Modify the depart function in order to return nothing as in the current CVS version it always returns true and adjust callers accordingly. - Fix the witness_addgraph() argument name prototype. - Remove unuseful code from itismychild(). This commit leads to a shrinked struct lock_object and so smaller locks, in particular on amd64 where 2 uintptr_t (16 bytes per-primitive) are gained. Reviewed by: jhb Notes: svn path=/head/; revision=179025
* Trylocks were implemented, remove stale commentKris Kennaway2008-04-121-2/+0
| | | | Notes: svn path=/head/; revision=178147
* Add rw_try_rlock() and rw_try_wlock() to rwlocks.Attilio Rao2008-04-011-0/+4
| | | | | | | | | | | | | These functions try the specified operation (rlocking and wlocking) and true is returned if the operation completes, false otherwise. The KPI is enriched by this commit, so __FreeBSD_version bumping and manpage updating will happen soon. Requested by: jeff, kris Notes: svn path=/head/; revision=177843
* Implement soon-to-be-used rw_unlock() macro.Pawel Jakub Dawidek2008-03-161-0/+6
| | | | Notes: svn path=/head/; revision=177260
* Adaptive spinning in write path with readers and writer starvation avoidance.Jeff Roberson2008-02-061-3/+6
| | | | | | | | | | | | | | | | | | | - Move recursion checking into rwlock inlines to free a bit for use with adaptive spinners. - Clear the RW_LOCK_WRITE_SPINNERS flag whenever the lock state changes causing write spinners to restart their loop. - Write spinners are limited by a count while readers hold the lock as there is no way to know for certain whether readers are running still. - In the read path block if there are write waiters or spinners to avoid starving writers. Use a new per-thread count, td_rw_rlocks, to skip starvation avoidance if it might cause a deadlock. - Remove or change invalid assertions in turnstiles. Reviewed by: attilio (developed parts of the patch as well) Sponsored by: Nokia Notes: svn path=/head/; revision=176017
* Fix some problems with lock profiling in rw locks:Attilio Rao2007-07-201-9/+6
| | | | | | | | | | | | | | | | | | | | - Adjust lock_profiling stubs semantic in the hard functions in order to be more accurate and trustable - As for sx locks, disable shared paths for lock_profiling. Actually, lock_profiling has a subtle race which makes results caming from shared paths not completely trustable. A macro stub (LOCK_PROFILING_SHARED) can be actually used for re-enabling this paths, but is currently intended for developing use only. - style(9) fixes Approved by: jeff, kmacy, jhb[1] Approved by: re [1] Had initial reservations not shared by others, conceded in the end. Notes: svn path=/head/; revision=171516
* Introduce a new rwlocks initialization function: rw_init_flags.Attilio Rao2007-06-261-3/+17
| | | | | | | | | | | | | | | | | | This is very similar to sx_init_flags: it initializes the rwlock using special flags passed as third argument (RW_DUPOK, RW_NOPROFILE, RW_NOWITNESS, RW_QUIET, RW_RECURSE). Among these, the most important new feature is probabilly that rwlocks can be acquired recursively now (for both shared and exclusive paths). Because of the recursion counter, the ABI is changed. Tested by: Timothy Redaelli <drizzt@gufi.org> Reviewed by: jhb Approved by: jeff (mentor) Approved by: re Notes: svn path=/head/; revision=171052
* Add destroyed cookie values for sx locks and rwlocks as well as extraJohn Baldwin2007-05-081-0/+1
| | | | | | | | KASSERTs so that any lock operations on a destroyed lock will panic or hang. Notes: svn path=/head/; revision=169394
* Fix a comment grammar nit.John Baldwin2007-03-271-1/+1
| | | | Notes: svn path=/head/; revision=167941
* Rename the 'mtx_object', 'rw_object', and 'sx_object' members of mutexes,John Baldwin2007-03-211-4/+4
| | | | | | | rwlocks, and sx locks to 'lock_object'. Notes: svn path=/head/; revision=167787
* Allow threads to atomically release rw and sx locks while waiting for anJohn Baldwin2007-03-091-0/+2
| | | | | | | | | | | | | | | | | | | | | | event. Locking primitives that support this (mtx, rw, and sx) now each include their own foo_sleep() routine. - Rename msleep() to _sleep() and change it's 'struct mtx' object to a 'struct lock_object' pointer. _sleep() uses the recently added lc_unlock() and lc_lock() function pointers for the lock class of the specified lock to release the lock while the thread is suspended. - Add wrappers around _sleep() for mutexes (mtx_sleep()), rw locks (rw_sleep()), and sx locks (sx_sleep()). msleep() still exists and is now identical to mtx_sleep(), but it is deprecated. - Rename SLEEPQ_MSLEEP to SLEEPQ_SLEEP. - Rewrite much of sleep.9 to not be msleep(9) centric. - Flesh out the 'RETURN VALUES' section in sleep.9 and add an 'ERRORS' section. - Add __nonnull(1) to _sleep() and msleep_spin() so that the compiler will warn if you try to pass a NULL wait channel. The functions already have a KASSERT to that effect. Notes: svn path=/head/; revision=167387
* Wrap a few lines at 80 cols.John Baldwin2007-03-071-2/+4
| | | | Notes: svn path=/head/; revision=167306
* Add rw_wowned() interface to rwlock(9), allowing a kernel thread toRobert Watson2007-02-261-0/+1
| | | | | | | | | | | determine if it holds an exclusive rwlock reference or not. This is non-ideal, but recursion scenarios in the network stack currently require it. Approved by: jhb Notes: svn path=/head/; revision=167024
* general LOCK_PROFILING cleanupKip Macy2007-02-261-3/+8
| | | | | | | | | | | | | | | - only collect timestamps when a lock is contested - this reduces the overhead of collecting profiles from 20x to 5x - remove unused function from subr_lock.c - generalize cnt_hold and cnt_lock statistics to be kept for all locks - NOTE: rwlock profiling generates invalid statistics (and most likely always has) someone familiar with that should review Notes: svn path=/head/; revision=167012
* Include pcpu.h so that kernel is buildable w/o RWLOCK_NOINLINE optionGleb Smirnoff2006-10-131-0/+1
| | | | | | | and with some code that uses rwlock(9). Notes: svn path=/head/; revision=163320
* Fix the name of the data set item for the SYSUNINIT in RW_SYSINIT to useJohn Baldwin2006-06-231-1/+1
| | | | | | | | | | 'rw' instead of 'mtx'. This should only be a cosmetic change rather than a functional one. Submitted by: Alex Lyashkov <shadow AT itt dot net dot ru> Notes: svn path=/head/; revision=159897
* Implement rw_try_upgrade() and rw_downgrade(). rw_try_upgrade() makes aJohn Baldwin2006-04-191-1/+5
| | | | | | | | single attempt at upgrading a read lock to a write lock, and rw_downgrade() converts curthread's write lock into a read lock. Notes: svn path=/head/; revision=157882
* Fix typo.Gleb Smirnoff2006-01-301-1/+1
| | | | Notes: svn path=/head/; revision=155042
* Add a basic reader/writer lock implementation to the kernel. ThisJohn Baldwin2006-01-271-0/+192
implementation is by no means perfect as far as some of the algorithms that it uses and the fact that it is missing some functionality (try locks and upgrades/downgrades are not there yet), however it does seem to work in my local testing. There is more detail in the comments in the code, but the short version follows. A reader/writer lock is very much like a regular mutex: it cannot be held across a voluntary sleep; it can be acquired in an interrupt thread; if the lock is held by a writer then the priority of any threads that block on the lock will be lent to the owner; the simple case lock operations all are done in a single atomic op. It also shares some similiarities with sx locks: it supports reader/writer semantics (multiple readers, but single writers); readers are allowed to recurse, but writers are not. We can extend this implementation further by either improving algorithms or adding new functionality, but this should at least give us a base to work with now. Reviewed by: arch (in theory) Tested on: i386 (4 cpu box with a kernel module that used 4 threads that randomly chose between read locks and write locks that ran w/o panicing for over a day solid. It usually panic'd within a few seconds when there were bugs during testing. :) The kernel module source is available on request.) Notes: svn path=/head/; revision=154941
generated by cgit v1.2.3 (git 2.25.1) at 2024-05-23 01:46:49 +0000