aboutsummaryrefslogtreecommitdiff
path: root/sys/sys/gtaskqueue.h
Commit message (Collapse)AuthorAgeFilesLines
* sys: Remove $FreeBSD$: two-line .h patternWarner Losh2023-08-161-2/+0
| | | | Remove /^\s*\*\n \*\s+\$FreeBSD\$$\n/
* spdx: The BSD-2-Clause-FreeBSD identifier is obsolete, drop -FreeBSDWarner Losh2023-05-121-1/+1
| | | | | | | | | The SPDX folks have obsoleted the BSD-2-Clause-FreeBSD identifier. Catch up to that fact and revert to their recommended match of BSD-2-Clause. Discussed with: pfg MFC After: 3 days Sponsored by: Netflix
* Import kernel WireGuard supportMatt Macy2020-11-291-0/+1
| | | | | | | | | | | | | Data path largely shared with the OpenBSD implementation by Matt Dunwoodie <ncon@nconroy.net> Reviewed by: grehan@freebsd.org MFC after: 1 month Sponsored by: Rubicon LLC, (Netgate) Differential Revision: https://reviews.freebsd.org/D26137 Notes: svn path=/head/; revision=368163
* Remove the "config" taskqgroup and its KPIs.Mark Johnston2020-03-301-3/+0
| | | | | | | | | | | Equivalent functionality is already provided by taskqueue(9), just use that instead. MFC after: 2 weeks Sponsored by: The FreeBSD Foundation Notes: svn path=/head/; revision=359438
* Simplify taskqgroup inititialization.Mark Johnston2020-03-301-11/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | taskqgroup initialization was broken into two steps: 1. allocate the taskqgroup structure, at SI_SUB_TASKQ; 2. initialize taskqueues, start taskqueue threads, enqueue "binder" tasks to bind threads to specific CPUs, at SI_SUB_SMP. Step 2 tries to handle the case where tasks have already been attached to a queue, by migrating them to their intended queue. In particular, tasks can't be enqueued before step 2 has completed. This breaks NFS mountroot on systems using an iflib-based driver when EARLY_AP_STARTUP is not defined, since mountroot happens before SI_SUB_SMP in this case. Simplify initialization: do all initialization except for CPU binding at SI_SUB_TASKQ. This means that until CPU binding is completed, group tasks may be executed on a CPU other than that to which they were bound, but this should not be a problem for existing users of the taskqgroup KPIs. Reported by: sbruno Tested by: bdragon, sbruno MFC after: 1 month Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D24188 Notes: svn path=/head/; revision=359436
* Add flag to struct task to mark the task as requiring network epoch.Gleb Smirnoff2020-02-111-5/+1
| | | | | | | | | | | | | | | | | | When processing a taskqueue and a task has associated epoch, then enter for duration of the task. If consecutive tasks belong to the same epoch, batch them. Now we are talking about the network epoch only. Shrink the ta_priority size to 8-bits. No current consumers use a priority that won't fit into 8 bits. Also complexity of taskqueue_enqueue() is a square of maximum value of priority, so we unlikely ever want to go over UCHAR_MAX here. Reviewed by: hselasky Differential Revision: https://reviews.freebsd.org/D23518 Notes: svn path=/head/; revision=357771
* Make taskqgroup_attach{,_cpu}(9) work across architecturesMarius Strobl2019-02-121-13/+29
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | So far, intr_{g,s}etaffinity(9) take a single int for identifying a device interrupt. This approach doesn't work on all architectures supported, as a single int isn't sufficient to globally specify a device interrupt. In particular, with multiple interrupt controllers in one system as found on e. g. arm and arm64 machines, an interrupt number as returned by rman_get_start(9) may be only unique relative to the bus and, thus, interrupt controller, a certain device hangs off from. In turn, this makes taskqgroup_attach{,_cpu}(9) and - internal to the gtaskqueue implementation - taskqgroup_attach_deferred{,_cpu}() not work across architectures. Yet in turn, iflib(4) as gtaskqueue consumer so far doesn't fit architectures where interrupt numbers aren't globally unique. However, at least for intr_setaffinity(..., CPU_WHICH_IRQ, ...) as employed by the gtaskqueue implementation to bind an interrupt to a particular CPU, using bus_bind_intr(9) instead is equivalent from a functional point of view, with bus_bind_intr(9) taking the device and interrupt resource arguments required for uniquely specifying a device interrupt. Thus, change the gtaskqueue implementation to employ bus_bind_intr(9) instead and intr_{g,s}etaffinity(9) to take the device and interrupt resource arguments required respectively. This change also moves struct grouptask from <sys/_task.h> to <sys/gtaskqueue.h> and wraps struct gtask along with the gtask_fn_t typedef into #ifdef _KERNEL as userland likes to include <sys/_task.h> or indirectly drags it in - for better or worse also with _KERNEL defined -, which with device_t and struct resource dependencies otherwise is no longer as easily possible now. The userland inclusion problem probably can be improved a bit by introducing a _WANT_TASK (as well as a _WANT_MOUNT) akin to the existing _WANT_PRISON etc., which is orthogonal to this change, though, and likely needs an exp-run. While at it: - Change the gt_cpu member in the grouptask structure to be of type int as used elswhere for specifying CPUs (an int16_t may be too narrow sooner or later), - move the gtaskqueue_enqueue_fn typedef from <sys/gtaskqueue.h> to the gtaskqueue implementation as it's only used and needed there, - change the GTASK_INIT macro to use "gtask" rather than "task" as argument given that it actually operates on a struct gtask rather than a struct task, and - let subr_gtaskqueue.c consistently use __func__ to print functions names. Reported by: mmel Reviewed by: mmel Differential Revision: https://reviews.freebsd.org/D19139 Notes: svn path=/head/; revision=344062
* Drain grouptaskqueue of the gtask before detaching it.Stephen Hurd2018-10-291-1/+4
| | | | | | | | | | | | | | | | | | taskqgroup_detach() would remove the task even if it was running or enqueued, which could lead to panics (see D17404). With this change, taskqgroup_detach() drains the task and sets a new flag which prevents the task from being scheduled again. I've added grouptask_block() and grouptask_unblock() to allow control over the flag from other locations as well. Reviewed by: Jeffrey Pieper <jeffrey.e.pieper@intel.com> MFC after: 1 week Sponsored by: Limelight Networks Differential Revision: https://reviews.freebsd.org/D17674 Notes: svn path=/head/; revision=339861
* iflib: mark irq allocation name parameter as constantStephen Hurd2018-05-291-2/+2
| | | | | | | | | | | | | | | | | The *name parameter passed to iflib_irq_alloc_generic and iflib_softirq_alloc_generic is never modified. Many places in code pass string literals and thus should not be modified. Mark the *name parameter as a const char * instead, so that we enforce that the name is not modified before passing to bus_describe_intr() Submitted by: Jacob Keller <jacob.e.keller@intel.com> Reviewed by: kmacy Sponsored by: Intel Corporation Differential Revision: https://reviews.freebsd.org/D15343 Notes: svn path=/head/; revision=334358
* Add taskqgroup_config_gtask_deinit to support teardown afterMatt Macy2018-05-091-0/+1
| | | | | | | | | taskqgroup_config_gtask_init. Approved by: sbruno Notes: svn path=/head/; revision=333426
* Separate list manipulation locking from state change in multicastStephen Hurd2018-05-021-1/+3
| | | | | | | | | | | | | | Multicast incorrectly calls in to drivers with a mutex held causing drivers to have to go through all manner of contortions to use a non sleepable lock. Serialize multicast updates instead. Submitted by: mmacy <mmacy@mattmacy.io> Reviewed by: shurd, sbruno Sponsored by: Limelight Networks Differential Revision: https://reviews.freebsd.org/D14969 Notes: svn path=/head/; revision=333175
* Update Matthew Macy contact infoStephen Hurd2017-12-191-1/+1
| | | | | | | | | | Email address has changed, uses consistent name (Matthew, not Matt) Reported by: Matthew Macy <mmacy@mattmacy.io> Differential Revision: https://reviews.freebsd.org/D13537 Notes: svn path=/head/; revision=326984
* SPDX: mostly fixes to previous changes.Pedro F. Giffuni2017-12-131-0/+2
| | | | | | | | Introduce the recently approved BSD-1-Clause and replace 0BSD which never did fit well our use cases. Notes: svn path=/head/; revision=326823
* Revert r323516 (iflib rollup)Stephen Hurd2017-09-161-26/+11
| | | | | | | | | | | | | | | This was really too big of a commit even if everything worked, but there are multiple new issues introduced in the one huge commit, so it's not worth keeping this until it's fixed. I'll work on splitting this up into logical chunks and introduce them one at a time over the next week or two. Approved by: sbruno (mentor) Sponsored by: Limelight Networks Notes: svn path=/head/; revision=323635
* Roll up iflib commits from github. This pulls in most of the work doneStephen Hurd2017-09-131-11/+26
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | by Matt Macy as well as other changes which he has accepted via pull request to his github repo at https://github.com/mattmacy/networking/ This should bring -CURRENT and the github repo into close enough sync to allow small feature branches rather than a large chain of interdependant patches being developed out of tree. The reset of the synchronization should be able to be completed on github by splitting the remaining changes that are not yet ready into short feature branches for later review as smaller commits. Here is a summary of changes included in this patch: 1) More checks when INVARIANTS are enabled for eariler problem detection 2) Group Task Queue cleanups - Fix use of duplicate shortdesc for gtaskqueue malloc type. Some interfaces such as memguard(9) use the short description to identify malloc types, so duplicates should be avoided. 3) Allow gtaskqueues to use ithreads in addition to taskqueues - In some cases, this can improve performance 4) Better logging when taskqgroup_attach*() fails to set interrupt affinity. 5) Do not start gtaskqueues until they're needed 6) Have mp_ring enqueue function enter the ABDICATED rather than BUSY state. This moves the TX to the gtaskq and allows processing to continue faster as well as make TX batching more likely. 7) Add an ift_txd_errata function to struct if_txrx. This allows drivers to inspect/modify mbufs before transmission. 8) Add a new IFLIB_NEED_ZERO_CSUM for drivers to indicate they need checksums zeroed for checksum offload to work. This avoids modifying packet data in the TX path when possible. 9) Use ithreads for iflib I/O instead of taskqueues 10) Clean up ioctl and support async ioctl functions 11) Prefetch two cachlines from each mbuf instead of one up to 128B. We often need to parse packet header info beyond 64B. 12) Fix potential memory corruption due to fence post error in bit_nclear() usage. 13) Improved hang detection and handling 14) If the packet is smaller than MTU, disable the TSO flags. This avoids extra packet parsing when not needed. 15) Move TCP header parsing inside the IS_TSO?() test. This avoids extra packet parsing when not needed. 16) Pass chains of mbufs that are not consumed by lro to if_input() rather call if_input() for each mbuf. 17) Re-arrange packet header loads to get as much work as possible done before a cache stall. 18) Lock the context when calling IFDI_ATTACH_PRE()/IFDI_ATTACH_POST()/ IFDI_DETACH(); 19) Attempt to distribute RX/TX tasks across cores more sensibly, especially when RX and TX share an interrupt. RX will attempt to take the first threads on a core, and TX will attempt to take successive threads. 20) Allow iflib_softirq_alloc_generic() to request affinity to the same cpus an interrupt has affinity with. This allows TX queues to ensure they are serviced by the socket the device is on. 21) Add new iflib sysctls to net.iflib: - timer_int - interval at which to run per-queue timers in ticks - force_busdma 22) Add new per-device iflib sysctls to dev.X.Y.iflib - rx_budget allows tuning the batch size on the RX path - watchdog_events Count of watchdog events seen since load 23) Fix error where netmap_rxq_init() could get called before IFDI_INIT() 24) e1000: Fixed version of r323008: post-cold sleep instead of DELAY when waiting for firmware - After interrupts are enabled, convert all waits to sleeps - Eliminates e1000 software/firmware synchronization busy waits after startup 25) e1000: Remove special case for budget=1 in em_txrx.c - Premature optimization which may actually be incorrect with multi-segment packets 26) e1000: Split out TX interrupt rather than share an interrupt for RX and TX. - Allows better performance by keeping RX and TX paths separate 27) e1000: Separate igb from em code where suitable Much easier to understand separate functions and "if (is_igb)" than previous tests like "if (reg_icr & (E1000_ICR_RXSEQ | E1000_ICR_LSC))" #blamebruno Reviewed by: sbruno Approved by: sbruno (mentor) Sponsored by: Limelight Networks Differential Revision: https://reviews.freebsd.org/D12235 Notes: svn path=/head/; revision=323516
* Move some other SI_SUB_INIT_IF initializations to SI_SUB_TASKQConrad Meyer2017-08-181-17/+1
| | | | | | | | | | | | Drop the EARLY_AP_STARTUP gtaskqueue code, as gtaskqueues are now initialized before APs are started. Reviewed by: hselasky@, jhb@ Sponsored by: Dell EMC Isilon Differential Revision: https://reviews.freebsd.org/D12054 Notes: svn path=/head/; revision=322672
* Make gtaskqueue compatible with drm-next such that they can be used with theSean Bruno2017-03-011-0/+1
| | | | | | | | | | linuxkpi tasklets. Submitted by: mmacy@nextbsd.org Reported by: hps Notes: svn path=/head/; revision=314502
* Shoot a couple of style bugs down in the macro declarations.Sean Bruno2017-01-261-5/+2
| | | | | | | Submitted by: bde Notes: svn path=/head/; revision=312814
* Resolve a non-fatal syntax warning left in by a trailing ;Sean Bruno2017-01-251-2/+2
| | | | | | | Submitted by: bde Notes: svn path=/head/; revision=312759
* iflib:Sean Bruno2017-01-241-16/+4
| | | | | | | | | | | | | | | | | | | Add internal tracking of smp startup status to reliably figure out what methods are to be used to get gtaskqueue up and running. e1000: Calculating this pointer gives undefined behaviour when (last == -1) (it is before the buffer). The pointer is always followed. Panics occurred when it points to an unmapped page. Otherwise, the pointed-to garbage tends to not have the E1000_TXD_STAT_DD bit set in it, so in the broken case the loop was usually null and the function just returned, and this was acidentally correct. Submitted by: bde Reported by: Matt Macy <mmacy@nextbsd.org> Notes: svn path=/head/; revision=312698
* Revert 312696 due to build tests.Sean Bruno2017-01-241-4/+16
| | | | Notes: svn path=/head/; revision=312697
* iflib:Sean Bruno2017-01-241-16/+4
| | | | | | | | | | | | | | | | | | | Add internal tracking of smp startup status to reliably figure out what methods are to be used to get gtaskqueue up and running. e1000: Calculating this pointer gives undefined behaviour when (last == -1) (it is before the buffer). The pointer is always followed. Panics occurred when it points to an unmapped page. Otherwise, the pointed-to garbage tends to not have the E1000_TXD_STAT_DD bit set in it, so in the broken case the loop was usually null and the function just returned, and this was acidentally correct. Submitted by: bde Reviewed by: Matt Macy <mmacy@nextbsd.org> Notes: svn path=/head/; revision=312696
* Adjust gtaskqueue startup again so that we catch the !SMP case andSean Bruno2017-01-191-6/+19
| | | | | | | | | | | | users that choose not to use EARLY_AP_STARTUP. There is still an initialization issue/panic with !SMP and !EARLY_AP_STARTUP that we have yet to resolve. Submitted by: bde Notes: svn path=/head/; revision=312434
* Change startup order for the no EARLY_AP_STARTUP case to initializeSean Bruno2017-01-161-1/+1
| | | | | | | | | | | | | gtaskqueue bits at SI_SUB_INIT_IF instead of waiting until SI_SUB_SMP which is far too late. Add an assertion in taskqgroup_attach() to catch startup initialization failures in the future. Reported by: kib bde Notes: svn path=/head/; revision=312293
* Fix a variety of cosmetic typos and misspellingsConrad Meyer2017-01-151-1/+1
| | | | | | | | | | | No functional change. PR: 216096, 216097, 216098, 216101, 216102, 216106, 216109, 216110 Reported by: Bulat <bltsrc at mail.ru> Sponsored by: Dell EMC Isilon Notes: svn path=/head/; revision=312235
* Update iflib to support more NIC designsStephen Hurd2016-08-121-0/+125
- Move group task queue into kern/subr_gtaskqueue.c - Change intr_enable to return an int so it can be detected if it's not implemented - Allow different TX/RX queues per set to be different sizes - Don't split up TX mbufs before transmit - Allow a completion queue for TX as well as RX - Pass the RX budget to isc_rxd_available() to allow an earlier return and avoid multiple calls Submitted by: shurd Reviewed by: gallatin Approved by: scottl Differential Revision: https://reviews.freebsd.org/D7393 Notes: svn path=/head/; revision=304021