aboutsummaryrefslogtreecommitdiff
path: root/sys/opencrypto
Commit message (Collapse)AuthorAgeFilesLines
* opencrypto: Add comments describing the new crypto_session layoutMark Johnston2021-01-201-0/+2
| | | | Requested by: rpokala
* opencrypto: Fix assignment of crypto completions to worker threadsMark Johnston2021-01-201-1/+4
| | | | | | | | | | | | | | | | | Since r336439 we simply take the session pointer value mod the number of worker threads (ncpu by default). On small systems this ends up funneling all completion work through a single thread, which becomes a bottleneck when processing IPSec traffic using hardware crypto drivers. (Software drivers such as aesni(4) are unaffected since they invoke completion handlers synchonously.) Instead, maintain an incrementing counter with a unique value per session, and use that to distribute work to completion threads. Reviewed by: cem, jhb MFC after: 2 weeks Sponsored by: Rubicon Communications, LLC ("Netgate") Differential Revision: https://reviews.freebsd.org/D28159
* opencrypto: Embed the driver softc in the session structureMark Johnston2021-01-201-13/+4
| | | | | | | | | | | | | | | Store the driver softc below the fields owned by opencrypto. This is a bit simpler and saves a pointer dereference when fetching the driver softc when processing a request. Get rid of the crypto session UMA zone. Session allocations are frequent or performance-critical enough to warrant a dedicated zone. No functional change intended. Reviewed by: cem, jhb Sponsored by: Rubicon Communications, LLC ("Netgate") Differential Revision: https://reviews.freebsd.org/D28158
* Remove the cloned file descriptors for /dev/crypto.John Baldwin2020-11-252-137/+86
| | | | | | | | | | | | | | | | | | | | | | | | | | Crypto file descriptors were added in the original OCF import as a way to provide per-open data (specifically the list of symmetric sessions). However, this gives a bit of a confusing API where one has to open /dev/crypto and then invoke an ioctl to obtain a second file descriptor. This also does not match the API used with /dev/crypto on other BSDs or with Linux's /dev/crypto driver. Character devices have gained support for per-open data via cdevpriv since OCF was imported, so use cdevpriv to simplify the userland API by permitting ioctls directly on /dev/crypto descriptors. To provide backwards compatibility, CRIOGET now opens another /dev/crypto descriptor via kern_openat() rather than dup'ing the existing file descriptor. This preserves prior semantics in case CRIOGET is invoked multiple times on a single file descriptor. Reviewed by: markj Relnotes: yes Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D27302 Notes: svn path=/head/; revision=368005
* Use void * in place of caddr_t.John Baldwin2020-11-061-14/+14
| | | | | | | | | Reviewed by: markj Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D27065 Notes: svn path=/head/; revision=367426
* Group session management routines together before first use.John Baldwin2020-11-061-104/+78
| | | | | | | | | | | | | | | - Rename cse*() to cse_*() to more closely match other local APIs in this file. - Merge the old csecreate() into cryptodev_create_session() and rename the new function to cse_create(). Reviewed by: markj Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D27070 Notes: svn path=/head/; revision=367425
* Move cryptof_ioctl() below the routines it calls.John Baldwin2020-11-061-195/+187
| | | | | | | | | Reviewed by: markj Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D27069 Notes: svn path=/head/; revision=367410
* Split logic to create new sessions into a separate function.John Baldwin2020-11-061-303/+300
| | | | | | | | | | | | This simplifies cryptof_ioctl as it now a wrapper around functions that contain the bulk of the per-ioctl logic. Reviewed by: markj Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D27068 Notes: svn path=/head/; revision=367409
* Move cryptodev_cb earlier before it is used.John Baldwin2020-11-051-19/+17
| | | | | | | | | | | | This is consistent with cryptodevkey_cb being defined before it is used and removes a prototype in the middle of the file. Reviewed by: markj Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D27067 Notes: svn path=/head/; revision=367407
* Style fixes for function prototypes and definitions.John Baldwin2020-11-052-37/+28
| | | | | | | | | Reviewed by: markj Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D27066 Notes: svn path=/head/; revision=367404
* Don't modify the destination pointer in ioctl requests.John Baldwin2020-11-051-6/+10
| | | | | | | | | | | | This breaks the case where the original pointer was NULL but an in-line IV was used. Reviewed by: markj Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D27064 Notes: svn path=/head/; revision=367403
* Replace some K&R function definitions with ANSI C.John Baldwin2020-11-032-13/+3
| | | | | | | | | Reviewed by: markj Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D27062 Notes: svn path=/head/; revision=367310
* Consistently use C99 fixed-width types in the in-kernel crypto code.John Baldwin2020-11-0316-119/+119
| | | | | | | | | Reviewed by: markj Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D27061 Notes: svn path=/head/; revision=367309
* opencrypto: Annotate hmac_init_(i|o)pad() to make auth_hash constMark Johnston2020-10-302-6/+6
| | | | | | | | | | | | This makes them friendlier to drivers that try to use const pointers whenever possible in their internal structures. Reviewed by: jhb Sponsored by: Rubicon Communications, LLC (Netgate) Differential Revision: https://reviews.freebsd.org/D26901 Notes: svn path=/head/; revision=367174
* Fix a couple of bugs for asym crypto introduced in r359374.John Baldwin2020-10-191-9/+12
| | | | | | | | | | | | | | | - Check for null pointers in the crypto_drivers[] array when checking for empty slots in crypto_select_kdriver(). - Handle the case where crypto_kdone() is invoked on a request where krq_cap is NULL due to not finding a matching driver. Reviewed by: markj Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D26811 Notes: svn path=/head/; revision=366852
* Mark asymmetric cryptography via OCF deprecated for 14.0.John Baldwin2020-10-192-0/+15
| | | | | | | | | | | | | | | | | Only one MIPS-specific driver implements support for one of the asymmetric operations. There are no in-kernel users besides /dev/crypto. The only known user of the /dev/crypto interface was the engine in OpenSSL releases before 1.1.0. 1.1.0 includes a rewritten engine that does not use the asymmetric operations due to lack of documentation. Reviewed by: cem, markj MFC after: 1 week Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D26810 Notes: svn path=/head/; revision=366844
* Add support for ESN in cryptosoftMarcin Wojtas2020-10-161-4/+7
| | | | | | | | | | | | | | | | | | | | | | | | | This patch adds support for IPsec ESN (Extended Sequence Numbers) in encrypt and authenticate mode (eg. AES-CBC and SHA256) and combined mode (eg. AES-GCM). For encrypt and authenticate mode the ESN is stored in separate crp_esn buffer because the high-order 32 bits of the sequence number are appended after the Next Header (RFC 4303). For combined modes the high-order 32 bits of the sequence number [e.g. RFC 4106, Chapter 5 AAD Construction] are part of crp_aad (prepared by netipsec layer in case of ESN support enabled), therefore non visible diff around combined modes. Submitted by: Grzegorz Jaszczyk <jaz@semihalf.com> Patryk Duda <pdk@semihalf.com> Reviewed by: jhb Differential revision: https://reviews.freebsd.org/D22364 Obtained from: Semihalf Sponsored by: Stormshield Notes: svn path=/head/; revision=366753
* Prepare crypto framework for IPsec ESN supportMarcin Wojtas2020-10-162-2/+6
| | | | | | | | | | | | | | | | | | | This permits requests (netipsec ESP and AH protocol) to provide the IPsec ESN (Extended Sequence Numbers) in a separate buffer. As with separate output buffer and separate AAD buffer not all drivers support this feature. Consumer must request use of this feature via new session flag. Submitted by: Grzegorz Jaszczyk <jaz@semihalf.com> Patryk Duda <pdk@semihalf.com> Reviewed by: jhb Differential revision: https://reviews.freebsd.org/D24838 Obtained from: Semihalf Sponsored by: Stormshield Notes: svn path=/head/; revision=366752
* Add support to the KTLS OCF module for AES-CBC MTE ciphersuites.John Baldwin2020-10-131-20/+273
| | | | | | | | | | | | | | This is a simplistic approach which encrypts each TLS record in two separate passes: one to generate the MAC and a second to encrypt. This supports TLS 1.0 connections with implicit IVs as well as TLS 1.1+ with explicit IVs. Reviewed by: gallatin Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D26730 Notes: svn path=/head/; revision=366677
* Simplify swcr_authcompute() after removal of deprecated algorithms.John Baldwin2020-10-061-29/+6
| | | | | | | | | | | | | | - Just use sw->octx != NULL to handle the HMAC case when finalizing the MAC. - Explicitly zero the on-stack auth context. Reviewed by: markj Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D26688 Notes: svn path=/head/; revision=366493
* Include sys/types.h hereWarner Losh2020-09-151-0/+1
| | | | | | | | | | | | It's included by header pollution in most of the compile environments. However, in the standalone envirnment, it's not included. Go ahead and include it always since the overhead is low and it is simpler that way. MFC After: 3 days Notes: svn path=/head/; revision=365751
* Name the on-stack union of compat thunks.John Baldwin2020-08-261-13/+14
| | | | | | | | | C does not permit an anonymous union at a top-level scope. Pointy hat to: jhb Notes: svn path=/head/; revision=364846
* Add freebsd32 compat support for CIOCCRYPTAEAD.John Baldwin2020-08-261-0/+61
| | | | | | | | | Reviewed by: markj (earlier version) Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D26179 Notes: svn path=/head/; revision=364840
* Simplify compat shims for /dev/crypto.John Baldwin2020-08-261-88/+112
| | | | | | | | | | | | | | | | | | - Make session handling always use the CIOGSESSION2 structure. CIOGSESSION requests use a thunk similar to COMPAT_FREEBSD32 session requests. This permits the ioctl handler to use the 'crid' field unconditionally. - Move COMPAT_FREEBSD32 handling out of the main ioctl handler body and instead do conversions in/out of thunk structures in dedicated blocks at the start and end of the ioctl function. Reviewed by: markj (earlier version) Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D26178 Notes: svn path=/head/; revision=364838
* crypto(9): add CRYPTO_BUF_VMPAGEAlan Somers2020-08-264-8/+285
| | | | | | | | | | | | | | | crypto(9) functions can now be used on buffers composed of an array of vm_page_t structures, such as those stored in an unmapped struct bio. It requires the running to kernel to support the direct memory map, so not all architectures can use it. Reviewed by: markj, kib, jhb, mjg, mat, bcr (manpages) MFC after: 1 week Sponsored by: Axcient Differential Revision: https://reviews.freebsd.org/D25671 Notes: svn path=/head/; revision=364799
* Add support for KTLS RX via software decryption.John Baldwin2020-07-231-5/+64
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Allow TLS records to be decrypted in the kernel after being received by a NIC. At a high level this is somewhat similar to software KTLS for the transmit path except in reverse. Protocols enqueue mbufs containing encrypted TLS records (or portions of records) into the tail of a socket buffer and the KTLS layer decrypts those records before returning them to userland applications. However, there is an important difference: - In the transmit case, the socket buffer is always a single "record" holding a chain of mbufs. Not-yet-encrypted mbufs are marked not ready (M_NOTREADY) and released to protocols for transmit by marking mbufs ready once their data is encrypted. - In the receive case, incoming (encrypted) data appended to the socket buffer is still a single stream of data from the protocol, but decrypted TLS records are stored as separate records in the socket buffer and read individually via recvmsg(). Initially I tried to make this work by marking incoming mbufs as M_NOTREADY, but there didn't seemed to be a non-gross way to deal with picking a portion of the mbuf chain and turning it into a new record in the socket buffer after decrypting the TLS record it contained (along with prepending a control message). Also, such mbufs would also need to be "pinned" in some way while they are being decrypted such that a concurrent sbcut() wouldn't free them out from under the thread performing decryption. As such, I settled on the following solution: - Socket buffers now contain an additional chain of mbufs (sb_mtls, sb_mtlstail, and sb_tlscc) containing encrypted mbufs appended by the protocol layer. These mbufs are still marked M_NOTREADY, but soreceive*() generally don't know about them (except that they will block waiting for data to be decrypted for a blocking read). - Each time a new mbuf is appended to this TLS mbuf chain, the socket buffer peeks at the TLS record header at the head of the chain to determine the encrypted record's length. If enough data is queued for the TLS record, the socket is placed on a per-CPU TLS workqueue (reusing the existing KTLS workqueues and worker threads). - The worker thread loops over the TLS mbuf chain decrypting records until it runs out of data. Each record is detached from the TLS mbuf chain while it is being decrypted to keep the mbufs "pinned". However, a new sb_dtlscc field tracks the character count of the detached record and sbcut()/sbdrop() is updated to account for the detached record. After the record is decrypted, the worker thread first checks to see if sbcut() dropped the record. If so, it is freed (can happen when a socket is closed with pending data). Otherwise, the header and trailer are stripped from the original mbufs, a control message is created holding the decrypted TLS header, and the decrypted TLS record is appended to the "normal" socket buffer chain. (Side note: the SBCHECK() infrastucture was very useful as I was able to add assertions there about the TLS chain that caught several bugs during development.) Tested by: rmacklem (various versions) Relnotes: yes Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D24628 Notes: svn path=/head/; revision=363464
* Consolidate duplicated code into a ktls_ocf_dispatch function.John Baldwin2020-07-231-52/+36
| | | | | | | | | | | This function manages the loop around crypto_dispatch and coordination with ktls_ocf_callback. Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D25757 Notes: svn path=/head/; revision=363461
* Don't dynamically allocate data structures for KTLS crypto requests.John Baldwin2020-07-201-70/+64
| | | | | | | | | | | | | | | Allocate iovec arrays and struct cryptop and struct ocf_operation objects on the stack to reduce avoid the overhead of malloc(). These structures are all small enough to fit on the stack of the KTLS worker threads. Reviewed by: gallatin Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D25692 Notes: svn path=/head/; revision=363379
* crypto(9): Stop checking for failures from malloc(M_WAITOK).Mark Johnston2020-07-201-5/+1
| | | | | | | | | | | PR: 240545 Submitted by: Andrew Reiter <arr@watson.org> Reviewed by: cem, delphij, jhb MFC after: 1 week Event: July 2020 Bugathon Notes: svn path=/head/; revision=363374
* Clean up crypto_init().Mark Johnston2020-07-171-31/+9
| | | | | | | | | | | | | | | | The function is called from a KLD load handler, so it may sleep. - Stop checking for errors from uma_zcreate(), they don't happen. - Convert M_NOWAIT allocations to M_WAITOK. - Remove error handling for existing M_WAITOK allocations. - Fix style. Reviewed by: cem, delphij, jhb MFC after: 1 week Differential Revision: https://reviews.freebsd.org/D25696 Notes: svn path=/head/; revision=363273
* Add crypto_initreq() and crypto_destroyreq().John Baldwin2020-07-162-6/+26
| | | | | | | | | | | | | These routines are similar to crypto_getreq() and crypto_freereq() but operate on caller-supplied storage instead of allocating crypto requests from a UMA zone. Reviewed by: markj Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D25691 Notes: svn path=/head/; revision=363262
* Convert cryptostats to a counter_u64 array.Mark Johnston2020-06-302-20/+42
| | | | | | | | | | | | The global counters were not SMP-friendly. Use per-CPU counters instead. Reviewed by: jhb Sponsored by: Rubicon Communications, LLC (Netgate) Differential Revision: https://reviews.freebsd.org/D25466 Notes: svn path=/head/; revision=362814
* Remove unused 32-bit compatibility structures from cryptodev.Mark Johnston2020-06-301-22/+0
| | | | | | | | | | | | The counters are exported by a sysctl and have the same width on all platforms anyway. Reviewed by: cem, delphij, jhb Sponsored by: Rubicon Communications, LLC (Netgate) Differential Revision: https://reviews.freebsd.org/D25465 Notes: svn path=/head/; revision=362802
* Remove CRYPTO_TIMING.Mark Johnston2020-06-302-94/+4
| | | | | | | | | | | | | It was added a very long time ago. It is single-threaded, so only really useful for basic measurements, and in the meantime we've gotten some more sophisticated profiling tools. Reviewed by: cem, delphij, jhb Sponsored by: Rubicon Communications, LLC (Netgate) Differential Revision: https://reviews.freebsd.org/D25464 Notes: svn path=/head/; revision=362801
* Zero the temporary HMAC key in hmac_init_pad().John Baldwin2020-06-251-0/+1
| | | | | | | | | Reviewed by: delphij Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25436 Notes: svn path=/head/; revision=362625
* Use zfree() instead of explicit_bzero() and free().John Baldwin2020-06-253-19/+4
| | | | | | | | | | | | | | | In addition to reducing lines of code, this also ensures that the full allocation is always zeroed avoiding possible bugs with incorrect lengths passed to explicit_bzero(). Suggested by: cem Reviewed by: cem, delphij Approved by: csprng (cem) Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25435 Notes: svn path=/head/; revision=362624
* Store the AAD in a separate buffer for KTLS.John Baldwin2020-06-231-63/+50
| | | | | | | | | | | | | | | For TLS 1.2 this permits reusing one of the existing iovecs without always having to duplicate both. While here, only duplicate the output iovec for TLS 1.3 if it will be used. Reviewed by: gallatin Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25291 Notes: svn path=/head/; revision=362523
* Add support to the crypto framework for separate AAD buffers.John Baldwin2020-06-224-49/+122
| | | | | | | | | | | | | | | | | | | | | | | | | | This permits requests to provide the AAD in a separate side buffer instead of as a region in the crypto request input buffer. This is useful when the main data buffer might not contain the full AAD (e.g. for TLS or IPsec with ESN). Unlike separate IVs which are constrained in size and stored in an array in struct cryptop, separate AAD is provided by the caller setting a new crp_aad pointer to the buffer. The caller must ensure the pointer remains valid and the buffer contents static until the request is completed (e.g. when the callback routine is invoked). As with separate output buffers, not all drivers support this feature. Consumers must request use of this feature via a new session flag. To aid in driver testing, kern.crypto.cryptodev_separate_aad can be set to force /dev/crypto requests to use a separate AAD buffer. Discussed with: cem Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25288 Notes: svn path=/head/; revision=362517
* Various optimizations to software AES-CCM and AES-GCM.John Baldwin2020-06-121-118/+205
| | | | | | | | | | | | | | | | | | | | | | | | | | | - Make use of cursors to avoid data copies for AES-CCM and AES-GCM. Pass pointers into the request's input and/or output buffers directly to the Update, encrypt, and decrypt hooks rather than always copying all data into a temporary block buffer on the stack. - Move handling for partial final blocks out of the main loop. This removes branches from the main loop and permits using encrypt/decrypt_last which avoids a memset to clear the rest of the block on the stack. - Shrink the on-stack buffers to assume AES block sizes and CCM/GCM tag lengths. - For AAD data, pass larger chunks to axf->Update. CCM can take each AAD segment in a single call. GMAC can take multiple blocks at a time. Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D25058 Notes: svn path=/head/; revision=362135
* Fix a regression in r361804 for TLS 1.3.John Baldwin2020-06-121-0/+1
| | | | | | | | | | I was not including the record type stored in the first byte of the trailer as part of the payload to be encrypted and hashed. Sponsored by: Netflix Notes: svn path=/head/; revision=362131
* Adjust crypto_apply function callbacks for OCF.John Baldwin2020-06-1016-110/+147
| | | | | | | | | | | | | | | | | | | | | | | | | | | - crypto_apply() is only used for reading a buffer to compute a digest, so change the data pointer to a const pointer. - To better match m_apply(), change the data pointer type to void * and the length from uint16_t to u_int. The length field in particular matters as none of the apply logic was splitting requests larger than UINT16_MAX. - Adjust the auth_xform Update callback to match the function prototype passed to crypto_apply() and crypto_apply_buf(). This removes the needs for casts when using the Update callback. - Change the Reinit and Setkey callbacks to also use a u_int length instead of uint16_t. - Update auth transforms for the changes. While here, use C99 initializers for auth_hash structures and avoid casts on callbacks. Reviewed by: cem Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25171 Notes: svn path=/head/; revision=362028
* Add a crypto capability flag for accelerated software drivers.John Baldwin2020-06-091-0/+1
| | | | | | | | | | | | | | | | | | Use this in GELI to print out a different message when accelerated software such as AESNI is used vs plain software crypto. While here, simplify the logic in GELI a bit for determing which type of crypto driver was chosen the first time by examining the capabilities of the matched driver after a single call to crypto_newsession rather than making separate calls with different flags. Reviewed by: delphij Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D25126 Notes: svn path=/head/; revision=361991
* Use separate output buffers for OCF requests in KTLS.John Baldwin2020-06-041-51/+110
| | | | | | | | | | | | | | | | | | KTLS encryption requests for file-backed data such as from sendfile(2) require the encrypted data to be stored in a separate buffer from the unencrypted file input data. Previously the OCF backend for KTLS manually copied the data from the input buffer to the output buffer before queueing the crypto request. Now the OCF backend will use a separate output buffer for such requests and avoid the copy. This mostly helps when an async co-processor is used by saving CPU cycles used on the copy. Reviewed by: gallatin (earlier version) Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D24545 Notes: svn path=/head/; revision=361804
* Add explicit bzero's of sensitive data in software crypto consumers.John Baldwin2020-06-031-23/+60
| | | | | | | | | | | Explicitly zero IVs, block buffers, and hashes/digests. Reviewed by: delphij Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D25057 Notes: svn path=/head/; revision=361773
* Increment the correct pointer when a crypto buffer spans an mbuf or iovec.John Baldwin2020-05-291-2/+2
| | | | | | | | | | | | | When a crypto_cursor_copyback() request spanned multiple mbufs or iovecs, the pointer into the mbuf/iovec was incremented instead of the pointer into the source buffer being copied from. PR: 246737 Reported by: Jenkins, ZFS test suite Sponsored by: Netflix Notes: svn path=/head/; revision=361617
* Add a sysctl knob to use separate output buffers for /dev/crypto.John Baldwin2020-05-251-33/+50
| | | | | | | | | | | This is a testing aid to permit using testing a driver's support of separate output buffers via cryptocheck. Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D24545 Notes: svn path=/head/; revision=361483
* Export the _kern_crypto sysctl node from crypto.c.John Baldwin2020-05-252-1/+5
| | | | | | | | Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D24545 Notes: svn path=/head/; revision=361482
* Add support for optional separate output buffers to in-kernel crypto.John Baldwin2020-05-256-339/+708
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Some crypto consumers such as GELI and KTLS for file-backed sendfile need to store their output in a separate buffer from the input. Currently these consumers copy the contents of the input buffer into the output buffer and queue an in-place crypto operation on the output buffer. Using a separate output buffer avoids this copy. - Create a new 'struct crypto_buffer' describing a crypto buffer containing a type and type-specific fields. crp_ilen is gone, instead buffers that use a flat kernel buffer have a cb_buf_len field for their length. The length of other buffer types is inferred from the backing store (e.g. uio_resid for a uio). Requests now have two such structures: crp_buf for the input buffer, and crp_obuf for the output buffer. - Consumers now use helper functions (crypto_use_*, e.g. crypto_use_mbuf()) to configure the input buffer. If an output buffer is not configured, the request still modifies the input buffer in-place. A consumer uses a second set of helper functions (crypto_use_output_*) to configure an output buffer. - Consumers must request support for separate output buffers when creating a crypto session via the CSP_F_SEPARATE_OUTPUT flag and are only permitted to queue a request with a separate output buffer on sessions with this flag set. Existing drivers already reject sessions with unknown flags, so this permits drivers to be modified to support this extension without requiring all drivers to change. - Several data-related functions now have matching versions that operate on an explicit buffer (e.g. crypto_apply_buf, crypto_contiguous_subsegment_buf, bus_dma_load_crp_buf). - Most of the existing data-related functions operate on the input buffer. However crypto_copyback always writes to the output buffer if a request uses a separate output buffer. - For the regions in input/output buffers, the following conventions are followed: - AAD and IV are always present in input only and their fields are offsets into the input buffer. - payload is always present in both buffers. If a request uses a separate output buffer, it must set a new crp_payload_start_output field to the offset of the payload in the output buffer. - digest is in the input buffer for verify operations, and in the output buffer for compute operations. crp_digest_start is relative to the appropriate buffer. - Add a crypto buffer cursor abstraction. This is a more general form of some bits in the cryptosoft driver that tried to always use uio's. However, compared to the original code, this avoids rewalking the uio iovec array for requests with multiple vectors. It also avoids allocate an iovec array for mbufs and populating it by instead walking the mbuf chain directly. - Update the cryptosoft(4) driver to support separate output buffers making use of the cursor abstraction. Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D24545 Notes: svn path=/head/; revision=361481
* Correct the minimum key length for Camellia to 16 bytes (128 bits).John Baldwin2020-05-221-1/+1
| | | | | | | MFC after: 1 week Notes: svn path=/head/; revision=361393
* Improve support for stream ciphers in the software encryption interface.John Baldwin2020-05-223-38/+78
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Add a 'native_blocksize' member to 'struct enc_xform' that ciphers can use if they support a partial final block. This is particular useful for stream ciphers, but can also apply to other ciphers. cryptosoft will only pass in native blocks to the encrypt and decrypt hooks. For the final partial block, 'struct enc_xform' now has new encrypt_last/decrypt_last hooks which accept the length of the final block. The multi_block methods are also retired. Mark AES-ICM (AES-CTR) as a stream cipher. This has some interesting effects on IPsec in that FreeBSD can now properly receive all packets sent by Linux when using AES-CTR, but FreeBSD can no longer interoperate with OpenBSD and older verisons of FreeBSD which assume AES-CTR packets have a payload padded to a 16-byte boundary. Kornel has offered to work on a patch to add a compatiblity sysctl to enforce additional padding for AES-CTR in esp_output to permit compatibility with OpenBSD and older versions of FreeBSD. AES-XTS continues to use a block size of a single AES block length. It is possible to adjust it to support partial final blocks by implementing cipher text stealing via encrypt_last/decrypt_last hooks, but I have not done so. Reviewed by: cem (earlier version) Tested by: Kornel Dulęba <mindal@semihalf.com> (AES-CTR with IPsec) Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D24906 Notes: svn path=/head/; revision=361390
generated by cgit v1.2.3 (git 2.25.1) at 2024-05-22 05:10:22 +0000