path: root/sys/sys/smr.h
Commit message (Collapse)AuthorAgeFilesLines
* Move SMR pointer type definition and access macros to smr_types.h.Mark Johnston2020-03-071-107/+2
| | | | | | | | | | | | | | | | | | | | | The intent is to provide a header that can be included by other headers without introducing too much pollution. smr.h depends on various headers and will likely grow over time, but is less likely to be required by system headers. Rename SMR_TYPE_DECLARE() to SMR_POINTER(): - One might use SMR to protect more than just pointers; it could be used for resizeable arrays, for example, so TYPE seems too generic. - It is useful to be able to define anonymous SMR-protected pointer types and the _DECLARE suffix makes that look wrong. Reviewed by: jeff, mjg, rlibby Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D23988 Notes: svn path=/head/; revision=358728
* Simplify lazy advance with a 64bit atomic cmpset.Jeff Roberson2020-02-271-4/+11
| | | | | | | | | | | | | | This provides the potential to force a lazy (tick based) SMR to advance when there are blocking waiters by decoupling the wr_seq value from the ticks value. Add some missing compiler barriers. Reviewed by: rlibby Differential Revision: https://reviews.freebsd.org/D23825 Notes: svn path=/head/; revision=358400
* Add an atomic-free tick moderated lazy update variant of SMR.Jeff Roberson2020-02-221-15/+78
| | | | | | | | | | | | | | | | | | | | | This enables very cheap read sections with free-to-use latencies and memory overhead similar to epoch. On a recent AMD platform a read section cost 1ns vs 5ns for the default SMR. On Xeon the numbers should be more like 1 ns vs 11. The memory consumption should be proportional to the product of the free rate and 2*1/hz while normal SMR consumption is proportional to the product of free rate and maximum read section time. While here refactor the code to make future additions more straightforward. Name the overall technique Global Unbound Sequences (GUS) and adjust some comments accordingly. This helps distinguish discussions of the general technique (SMR) vs this specific implementation (GUS). Discussed with: rlibby, markj Notes: svn path=/head/; revision=358236
* Since r357940 it is no longer possible to use a single type cast for allJeff Roberson2020-02-191-2/+2
| | | | | | | atomic_*_ptr functions. Notes: svn path=/head/; revision=358129
* Type validating smr protected pointer accessors.Jeff Roberson2020-02-191-0/+92
| | | | | | | | | | | | | | This API is intended to provide some measure of safety with SMR protected pointers. A struct wrapper provides type checking and a guarantee that all access is mediated by the API unless abused. All modifying functions take an assert as an argument to guarantee that the required synchronization is present. Reviewed by: kib, markj, mjg Differential Revision: https://reviews.freebsd.org/D23711 Notes: svn path=/head/; revision=358096
* Add more precise SMR entry asserts.Jeff Roberson2020-02-131-0/+9
| | | | Notes: svn path=/head/; revision=357882
* smr.h: fix build after r357641Ryan Libby2020-02-071-0/+1
| | | | | | | | | | | | r357641 missed committing the change to sys/sys/smr.h. Reported by: jkim Submitted by: jeff Reviewed by: rlibby Differential Revision: https://reviews.freebsd.org/D23464 Notes: svn path=/head/; revision=357644
* Implement a deferred write advancement feature that can be used to furtherJeff Roberson2020-02-041-0/+8
| | | | | | | | | | amortize shared cacheline writes. Discussed with: rlibby Differential Revision: https://reviews.freebsd.org/D23462 Notes: svn path=/head/; revision=357487
* Add two missing fences with comments describing them. These were found byJeff Roberson2020-01-311-2/+9
| | | | | | | | | | | inspection and after a lengthy discussion with jhb and kib. They have not produced test failures. Don't pointer chase through cpu0's smr. Use cpu correct smr even when not in a critical section to reduce the likelihood of false sharing. Notes: svn path=/head/; revision=357355
* Don't use "All rights reserved" in new copyrights.Jeff Roberson2020-01-311-2/+1
| | | | | | | Requested by: rgrimes Notes: svn path=/head/; revision=357316
* Implement a safe memory reclamation feature that is tightly coupled with UMA.Jeff Roberson2020-01-311-0/+180
This is in the same family of algorithms as Epoch/QSBR/RCU/PARSEC but is a unique algorithm. This has 3x the performance of epoch in a write heavy workload with less than half of the read side cost. The memory overhead is significantly lessened by limiting the free-to-use latency. A synthetic test uses 1/20th of the memory vs Epoch. There is significant further discussion in the comments and code review. This code should be considered experimental. I will write a man page after it has settled. After further validation the VM will begin using this feature to permit lockless page lookups. Both markj and cperciva tested on arm64 at large core counts to verify fences on weaker ordering architectures. I will commit a stress testing tool in a follow-up. Reviewed by: mmacy, markj, rlibby, hselasky Discussed with: sbahara Differential Revision: https://reviews.freebsd.org/D22586 Notes: svn path=/head/; revision=357314