aboutsummaryrefslogtreecommitdiff
path: root/sys/kern/kern_timeout.c
Commit message (Collapse)AuthorAgeFilesLines
* callout: retire callout_async_drain()Gleb Smirnoff2024-01-241-32/+5
| | | | | | | | | | | | | | This function was used only in TCP before 446ccdd08e2a. It was born in pain in 2016 to plug different complex panics in TCP timers. It wasn't warmly accepted in phabricator by all of the reviewers and my recollection of overall agreement was that "if you need this KPI, then you'd better fix your code to not need it". However, the function served its duty well all the way to FreeBSD 14. But now that TCP doesn't need it anymore, let's retire it to reduce complexity of callout code and also to avoid its further use. Reviewed by: jhb, markj, kib, rrs Differential Revision: https://reviews.freebsd.org/D43546
* sys: Remove ancient SCCS tags.Warner Losh2023-11-271-2/+0
| | | | | | | | Remove ancient SCCS tags from the tree, automated scripting, with two minor fixup to keep things compiling. All the common forms in the tree were removed with a perl script. Sponsored by: Netflix
* sys: Remove $FreeBSD$: one-line .c patternWarner Losh2023-08-161-2/+0
| | | | Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/
* callout: Remove an unneeded MTX_NEWMark Johnston2023-04-261-1/+1
| | | | | Reported by: hselasky Fixes: 78cfa762ebf2 ("callout: Move per-CPU callout state into the dpcpu region")
* callout: Move per-CPU callout state into the dpcpu regionMark Johnston2023-04-261-3/+3
| | | | | | | | | | | | This eliminates some static bloat in amd64 kernels and reduces the penalty of increasing MAXCPU. The structures now also maintain NUMA affinity. No functional change intended. PR: 269572 Reviewed by: mjg, kib MFC after: 2 weeks Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D39807
* callout(9): adopt old references to timeout(9)Mitchell Horne2023-03-201-1/+1
| | | | | | | | | | | | | timeout(9) was removed a couple of years ago; all consumers now use the callout(9) interface. Explicitly do not bump .Dd anywhere, as this is not a content or semantic change. Reviewed by: markj, jhb, Pau Amma <pauamma@gundo.com> MFC after: 1 week Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D39136
* ddb: annotate some commands with DB_CMD_MEMSAFEMitchell Horne2022-07-181-1/+1
| | | | | | | | | | This is not completely exhaustive, but covers a large majority of commands in the tree. Reviewed by: markj Sponsored by: Juniper Networks, Inc. Sponsored by: Klara, Inc. Differential Revision: https://reviews.freebsd.org/D35583
* callout: Simplify the inner loop in callout_process() a bitMark Johnston2022-07-171-34/+31
| | | | | | | | | | - Use LIST_FOREACH_SAFE. - Simplify control flow. No functional change intended. MFC after: 1 week Sponsored by: The FreeBSD Foundation
* callout: Remove a redundant parameter to callout_cc_add()Mark Johnston2022-07-171-4/+4
| | | | | | | | The passed cpuid is always equal to the one stored in the callout structure. No functional change intended. MFC after: 1 week Sponsored by: The FreeBSD Foundation
* ithreads: Support priority adjustment by schedulers.John Baldwin2022-07-141-1/+1
| | | | | | | | | | | Use sched_wakeup instead of sched_add when marking an ithread runnable. This allows schedulers to reset their internal time slice tracking state and restore the base ithread priority when an ithread resumes from idle. Reviewed by: markj Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D35643
* Add sched_ithread_prio to set the base priority of an interrupt thread.John Baldwin2022-07-141-1/+1
| | | | | | | | | Use it instead of sched_prio when setting the priority of an interrupt thread. Reviewed by: kib, markj Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D35642
* callout: Simplify cpuid validation in callout_reset_sbt_on()Mark Johnston2022-07-131-15/+4
| | | | | | | | | | | | | | - Remove a flag variable. - Convert a runtime check of the passed cpuid to a KASSERT. - Remove the cc_inited flag. An attempt to schedule a callout before SI_SUB_CPU will crash anyway since the per-CPU mutexes won't have been initialized, and that flag was only checked in the case where a cpuid was explicitly specified by the caller. No functional change intended. MFC after: 2 weeks Sponsored by: The FreeBSD Foundation
* Run softclock threads at a hardware ithread priority.John Baldwin2022-04-211-1/+1
| | | | | | | | | Add a new PI_SOFTCLOCK for use by softclock threads. Currently this maps to PI_AV which is the second-highest ithread priority. Reviewed by: mav, kib Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D33693
* callout: fix using shared rmlocksKristof Provost2022-04-201-1/+1
| | | | | | | | | | | | | 15b1eb142c changed the callout code to store the CALLOUT_SHAREDLOCK flag in c_iflags (where it used to be c_flags), but failed to update the check in softclock_call_cc(). This resulted in the callout code always taking the write lock, even if a read lock had been requested (with the CALLOUT_SHAREDLOCK flag in callout_init_rm()). Reviewed by: markj MFC after: 1 week Sponsored by: Rubicon Communications, LLC ("Netgate") Differential Revision: https://reviews.freebsd.org/D34959
* callout: Remove the CS_EXECUTING flagMark Johnston2022-03-231-2/+2
| | | | | | | | It is now unused. MFC after: 1 month Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D34626
* kern: harvest entropy from calloutsKyle Evans2022-02-031-0/+11
| | | | | | | | | | | | | | | | | | | 74cf7cae4d22 ("softclock: Use dedicated ithreads for running callouts.") switched callouts away from the swi infrastructure. It turns out that this was a major source of entropy in early boot, which we've now lost. As a result, first boot on hardware without a 'fast' entropy source would block waiting for fortuna to be seeded with little hope of progressing without manual intervention. Let's resolve it by explicitly harvesting entropy in callout_process() if we've handled any callouts. cc/curthread/now seem to be reasonable sources of entropy, so use those. Discussed with: jhb (also proposed initial patch) Reported by: many Reviewed by: cem, markm (both csprng) Differential Revision: https://reviews.freebsd.org/D34150
* Reduce minimum idle hardclock rate from 2Hz to 1Hz.Alexander Motin2022-01-101-3/+2
| | | | | | | On idle 80-thread system it allows to improve package-level idle state residency and so power consumption by several percent. MFC after: 2 weeks
* callout: Wait for the softclock thread to switch before reschedulingMark Johnston2021-12-311-0/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When a softclock thread prepares to go off-CPU, the following happens in the context of the thread: 1. callout state is locked 2. thread state is set to IWAIT 3. thread lock is switched from the tdq lock to the callout lock 4. tdq lock is released 5. sched_switch() sets td_lock to &blocked_lock 6. sched_switch() releases old td_lock (callout lock) 7. sched_switch() removes td from its runqueue 8. cpu_switch() sets td_lock back to the callout lock Suppose a timer interrupt fires while the softclock thread is switching off, and callout_process() schedules the softclock thread. Then there is a window between steps 5 and 8 where callout_process() can call sched_add() while td_lock is &blocked_lock, but this is not correct since the thread is not logically locked. callout_process() thus needs to spin waiting for the softclock thread to finish switching off (i.e., after step 8 completes) before rescheduling it, since callout_process() does not acquire the thread lock directly. Reported by: syzbot+fb44dbf6734ff492c337@syzkaller.appspotmail.com Fixes: 74cf7cae4d22 ("softclock: Use dedicated ithreads for running callouts.") Reviewed by: mav, kib, jhb Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D33709
* softclock: Use dedicated ithreads for running callouts.John Baldwin2021-12-301-39/+76
| | | | | | | | | | | Rather than using the swi infrastructure, rewrite softclock() as a thread loop (softclock_thread()) and use it as the main routine of the softclock threads. The threads use the CC_LOCK as the thread lock when idle. Reviewed by: mav, imp, kib Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D33683
* callout(9): Allow spin locks use with callout_init_mtx().Alexander Motin2021-09-031-5/+5
| | | | | | | | | | | | | | | Implement lock_spin()/unlock_spin() lock class methods, moving the assertion to _sleep() instead. Change assertions in callout(9) to allow spin locks for both regular and C_DIRECT_EXEC cases. In case of C_DIRECT_EXEC callouts spin locks are the only locks allowed actually. As the first use case allow taskqueue_enqueue_timeout() use on fast task queues. It actually becomes more efficient due to avoided extra context switches in callout(9) thanks to C_DIRECT_EXEC. MFC after: 2 weeks Reviewed by: hselasky Differential Revision: https://reviews.freebsd.org/D31778
* callout: Make cc_cpu local to kern_timeout.cMark Johnston2021-07-161-4/+4
| | | | | | | No functional change intended. MFC after: 1 week Sponsored by: The FreeBSD Foundation
* Balance parentheses in sysctl descriptionsPiotr Pawel Stefaniak2021-04-111-1/+1
|
* callout(9): Remove some leftover APM BIOS supportMark Johnston2020-11-271-65/+0
| | | | | | | | | | | This code is obsolete since r366546. Reviewed by: imp Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D27267 Notes: svn path=/head/; revision=368112
* callout(9): Fix a race between CPU migration and callout_drain()Mark Johnston2020-11-191-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Suppose a running callout re-arms itself, and before the callout finishes running another CPU calls callout_drain() and goes to sleep. softclock_call_cc() will wake up the draining thread, which may not run immediately if there is a lot of CPU load. Furthermore, the callout is still in the callout wheel so it can continue to run and re-arm itself. Then, suppose that the callout migrates to another CPU before the draining thread gets a chance to run. The draining thread is in this loop in _callout_stop_safe(): while (cc_exec_curr(cc) == c) { CC_UNLOCK(cc); sleep(); CC_LOCK(cc); } but after the migration, cc points to the wrong CPU's callout state. Then the draining thread goes off and removes the callout from the wheel, but does so using the wrong lock and per-CPU callout state. Fix the problem by doing a re-lookup of the callout CPU after sleeping. Reported by: syzbot+79569cd4d76636b2cc1c@syzkaller.appspotmail.com Reported by: syzbot+1b27e0237aa22d8adffa@syzkaller.appspotmail.com Reported by: syzbot+e21aa5b85a9aff90ef3e@syzkaller.appspotmail.com Reviewed by: emaste, hselasky Tested by: pho MFC after: 1 week Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D27266 Notes: svn path=/head/; revision=367849
* Remove NO_EVENTTIMERS supportMark Johnston2020-11-191-19/+3
| | | | | | | | | | | | | | The arm configs that required it have been removed from the tree. Removing this option makes the callout code easier to read and discourages developers from adding new configs without eventtimer drivers. Reviewed by: ian, imp, mav Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D27270 Notes: svn path=/head/; revision=367826
* Assert that cc_exec_drain(cc, direct) is NULL before assigning a new value.Hans Petter Selasky2020-09-021-0/+6
| | | | | | | | | | Suggested by: markj@ Tested by: callout_test MFC after: 1 week Sponsored by: Mellanox Technologies // NVIDIA Networking Notes: svn path=/head/; revision=365238
* Micro optimise _callout_stop_safe() by removing dead code.Hans Petter Selasky2020-09-021-5/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The CS_DRAIN flag cannot be set at the same time like the async-drain function pointer is set. These are orthogonal features. Assert this at the beginning of the function. Before: if (flags & CS_DRAIN) { /* FALLTHROUGH */ } else if (xxx) { return yyy; } if (drain) { zzz = drain; } After: if (flags & CS_DRAIN) { /* FALLTHROUGH */ } else if (xxx) { return yyy; } else { if (drain) { zzz = drain; } } Reviewed by: markj@ Tested by: callout_test Differential Revision: https://reviews.freebsd.org/D26285 MFC after: 1 week Sponsored by: Mellanox Technologies // NVIDIA Networking Notes: svn path=/head/; revision=365237
* kern: clean up empty lines in .c and .h filesMateusz Guzik2020-09-011-1/+1
| | | | Notes: svn path=/head/; revision=365222
* Remove misleading / redundant bzero in callout_callwheel_initConrad Meyer2020-03-161-1/+0
| | | | | | | | | | | | | | | | | | | | | | | | | The intent seems to be zeroing all of the cc_cpu array, or its singleton on such platforms. The assumption made is that the BSP is always zero. The code smell was introduced in r326218, which changed the prior explicit zero to 'curcpu'. The change is only valid if curcpu continues to be zero, contrary to the aim expressed in that commit message. So, more succinctly, the expression could be: memset(cc_cpu,0,sizeof(cc_cpu)). However, there's no point. cc_cpu lives in the data section and has a zero initial value already. So this revision just removes the problematic statement. No functional change. Appeases a (false positive, ish) Coverity CID. CID: 1383567 Reported by: Puneeth Jothaiah <puneethkumar.jothaia AT dell.com> Reviewed by: kib Sponsored by: Dell EMC Isilon Differential Revision: https://reviews.freebsd.org/D24089 Notes: svn path=/head/; revision=359022
* Remove the deprecated timeout(9) interface.John Baldwin2019-12-131-155/+23
| | | | | | | | | | All in-tree consumers have been converted to callout(9). Reviewed by: kib, markj Differential Revision: https://reviews.freebsd.org/D22602 Notes: svn path=/head/; revision=355732
* Add a callout_func_t typedef for functions used with callout_*().John Baldwin2019-12-101-9/+7
| | | | | | | | | | | | | | | | | This typedef is the same as timeout_t except that it is in the callout namespace and header. Use this typedef in various places of the callout implementation that were either using the raw type or timeout_t. While here, add <sys/callout.h> to the manpage. Reviewed by: kib, imp MFC after: 1 month Differential Revision: https://reviews.freebsd.org/D22751 Notes: svn path=/head/; revision=355600
* Mark some more hot global variables with __read_mostly.Alexander Motin2019-12-041-2/+3
| | | | | | | MFC after: 1 week Notes: svn path=/head/; revision=355404
* cc_ktr_event_name is used only with KTRGleb Smirnoff2019-11-211-0/+4
| | | | Notes: svn path=/head/; revision=354988
* Allocate callout wheel from the respective memory domain.Alexander Motin2019-09-211-2/+4
| | | | | | | MFC after: 1 week Notes: svn path=/head/; revision=352579
* Save the last callout function executed on each CPUEric van Gyzen2019-07-031-1/+46
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Save the last callout function pointer (and its argument) executed on each CPU for inspection by a debugger. Add a ddb `show callout_last` command to show these pointers. Add a kernel module that I used for testing that command. Relocate `ce_migration_cpu` to reduce padding and therefore preserve the size of `struct callout_cpu` (320 bytes on amd64) despite the added members. This should help diagnose reference-after-free bugs where the callout's mutex has already been freed when `softclock_call_cc` tries to unlock it. You might hope that the pointer would still be available, but it isn't. The argument to that function is on the stack (because `softclock_call_cc` uses it later), and that might be enough in some cases, but even then, it's very laborious. A pointer to the callout is saved right before these newly added fields, but that callout might have been freed. We still have the pointer to its associated mutex, and the name within might be enough, but it might also have been freed. Reviewed by: markj jhb MFC after: 2 weeks Sponsored by: Dell EMC Isilon Differential Revision: https://reviews.freebsd.org/D20794 Notes: svn path=/head/; revision=349677
* Remove some, but not all, assumptions that the BSP is CPU 0 and that CPUsNathan Whitehorn2017-11-251-5/+5
| | | | | | | | | are numbered densely from there to n_cpus. MFC after: 1 month Notes: svn path=/head/; revision=326218
* sys: further adoption of SPDX licensing ID tags.Pedro F. Giffuni2017-11-201-0/+2
| | | | | | | | | | | | | | | | | Mainly focus on files that use BSD 3-Clause license. The Software Package Data Exchange (SPDX) group provides a specification to make it easier for automated tools to detect and summarize well known opensource licenses. We are gradually adopting the specification, noting that the tags are considered only advisory and do not, in any way, superceed or replace the license texts. Special thanks to Wind River for providing access to "The Duke of Highlander" tool: an older (2014) run over FreeBSD tree was useful as a starting point. Notes: svn path=/head/; revision=326023
* Remove register keyword from sys/ and ANSIfy prototypesEd Maste2017-05-171-1/+1
| | | | | | | | | | | | | | | A long long time ago the register keyword told the compiler to store the corresponding variable in a CPU register, but it is not relevant for any compiler used in the FreeBSD world today. ANSIfy related prototypes while here. Reviewed by: cem, jhb Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D10193 Notes: svn path=/head/; revision=318389
* When draining a callout, don't clear CALLOUT_ACTIVE while it is running.Mark Johnston2017-03-151-2/+6
| | | | | | | | | | | | | The callout may reschedule itself and execute again before callout_drain() returns, but we should not clear CALLOUT_ACTIVE until the callout is stopped. Tested by: pho MFC after: 2 weeks Sponsored by: Dell EMC Isilon Notes: svn path=/head/; revision=315289
* Permit timed sleeps for threads other than thread0 before timers are working.John Baldwin2016-11-251-0/+2
| | | | | | | | | | | | | | | | | The callout subsystem already handles early callouts and schedules the first clock interrupt appropriately based on the currently pending callouts. The one nit to fix was that callouts scheduled via C_HARDCLOCK during early boot could fire too early once timers were enabled as the per-CPU base time is always zero until timers are initialized. The change in callout_when() handles this case by using the current uptime as the base time of the callout during bootup if the per-CPU base time is zero. Reviewed by: kib MFC after: 2 weeks Sponsored by: Netflix Notes: svn path=/head/; revision=309148
* Renumber license clauses in sys/kern to avoid skipping #3Ed Maste2016-09-151-1/+1
| | | | Notes: svn path=/head/; revision=305832
* Fix a stupid typo (or copy/paste buffer malfunction).Gleb Smirnoff2016-08-161-1/+1
| | | | Notes: svn path=/head/; revision=304245
* We should not be allowing a timeout to reset when a drain is in progress onGleb Smirnoff2016-08-161-1/+1
| | | | | | | | | | | | | it (either async or sync drain). At this moment the only user of drain is TCP, but TCP wouldn't reschedule a callout after it has drained it, since it drains only when a tcpcb is closed. This for now the problem isn't observed. Submitted by: rrs Notes: svn path=/head/; revision=304244
* Fix indentation.Konstantin Belousov2016-08-101-1/+1
| | | | | | | | Reported by: hselasky MFC after: 17 days Notes: svn path=/head/; revision=303919
* Extract the calculation of the callout fire time into the new functionKonstantin Belousov2016-07-281-42/+53
| | | | | | | | | | | | | | callout_when(9). See the man page update for the description of the intended use. Tested by: pho Reviewed by: jhb, bjk (man page updates) Sponsored by: The FreeBSD Foundation MFC after: 1 month X-Differential revision: https://reviews.freebsd.org/D7137 Notes: svn path=/head/; revision=303425
* Redo the r302894: the very new value for a non-scheduled callout is -1.Gleb Smirnoff2016-07-201-1/+7
| | | | | | | | | | | This was recently added in r290664. Noticed by: hselasky Tested by: Larry Rosenman <ler lerctr.org> PR: 210884 Notes: svn path=/head/; revision=303098
* Revert r303037. It re-introduces the panic with TCP timers.Gleb Smirnoff2016-07-201-44/+22
| | | | | | | Agreed by: rrs, re (gjb) Notes: svn path=/head/; revision=303096
* This reverts out Gleb's changes and adds three smallRandall Stewart2016-07-191-22/+44
| | | | | | | | | | | fixes that I think closes up the races Gleb was looking for. This is running quite nicely in Netflix and now no longer causes TCP-tcb leaks. Differential Revision: 7135 Notes: svn path=/head/; revision=303037
* Revert the last commit. It must get more review and testing first.Gleb Smirnoff2016-07-181-1/+1
| | | | Notes: svn path=/head/; revision=302998
* Redo the r302894: the very new value for a non-scheduled callout is -1.Gleb Smirnoff2016-07-181-1/+1
| | | | | | | | | | This was recently added in r290664. Noticed by: hselasky PR: 210884 Notes: svn path=/head/; revision=302997
generated by cgit v1.2.3 (git 2.25.1) at 2024-04-26 04:07:30 +0000