diff options
| author | Mark Johnston <markj@FreeBSD.org> | 2021-01-20 01:34:35 +0000 |
|---|---|---|
| committer | Mark Johnston <markj@FreeBSD.org> | 2021-02-03 14:37:17 +0000 |
| commit | 10ed8ab4ab9bd0239f2913ac2f35af9b9f76221d (patch) | |
| tree | 214160125be3a6cb5656db0203dafd982aad93d8 | |
| parent | 71abbe15d01f73a4e55a732f21180d339eab631d (diff) | |
| download | src-10ed8ab4ab9bd0239f2913ac2f35af9b9f76221d.tar.gz src-10ed8ab4ab9bd0239f2913ac2f35af9b9f76221d.zip | |
opencrypto: Fix assignment of crypto completions to worker threads
Since r336439 we simply take the session pointer value mod the number of
worker threads (ncpu by default). On small systems this ends up
funneling all completion work through a single thread, which becomes a
bottleneck when processing IPSec traffic using hardware crypto drivers.
(Software drivers such as aesni(4) are unaffected since they invoke
completion handlers synchonously.)
Instead, maintain an incrementing counter with a unique value per
session, and use that to distribute work to completion threads.
Reviewed by: cem, jhb
Sponsored by: Rubicon Communications, LLC ("Netgate")
Differential Revision: https://reviews.freebsd.org/D28159
(cherry picked from commit 98d788c867b9e1d7a7e290254443b87ea77d8ab1)
| -rw-r--r-- | sys/opencrypto/crypto.c | 5 |
1 files changed, 4 insertions, 1 deletions
diff --git a/sys/opencrypto/crypto.c b/sys/opencrypto/crypto.c index dfd22662e87f..bc23056e86ad 100644 --- a/sys/opencrypto/crypto.c +++ b/sys/opencrypto/crypto.c @@ -94,6 +94,7 @@ struct crypto_session { void *softc; uint32_t hid; uint32_t capabilities; + uint64_t id; }; SDT_PROVIDER_DEFINE(opencrypto); @@ -572,6 +573,7 @@ again: int crypto_newsession(crypto_session_t *cses, struct cryptoini *cri, int crid) { + static uint64_t sessid = 0; crypto_session_t res; void *softc_mem; struct cryptocap *cap; @@ -616,6 +618,7 @@ restart: softc_mem = malloc(softc_size, M_CRYPTO_DATA, M_WAITOK | M_ZERO); res = uma_zalloc(cryptoses_zone, M_WAITOK | M_ZERO); res->softc = softc_mem; + res->id = atomic_fetchadd_64(&sessid, 1); CRYPTO_DRIVER_LOCK(); cap = crypto_checkdriver(hid); @@ -1016,7 +1019,7 @@ crypto_dispatch(struct cryptop *crp) binuptime(&crp->crp_tstamp); #endif - crp->crp_retw_id = ((uintptr_t)crp->crp_session) % crypto_workers_num; + crp->crp_retw_id = crp->crp_session->id % crypto_workers_num; if (CRYPTOP_ASYNC(crp)) { if (crp->crp_flags & CRYPTO_F_ASYNC_KEEPORDER) { |