I'm pretty sure this never worked reliably based on code inspection,
and it's unlikely to have ever been tested because it only applies
when unregistering a driver -- but we have no crypto drivers for
removable devices, so it would only apply if we went out of our way
to trigger detach with drvctl.
Instead, just make the operation fail with ENODEV, and remove all the
callback logic to resubmit the request on EAGAIN. (Maybe this should
be ENXIO, but crypto_kdispatch already does ENODEV.)
Previously, crypto_newsession could sometimes return 0 as the
driver-specific part of the session id, and 0 as the hid, for sid=0.
But netipsec assumes that it is always safe to free sid=0 from
zero-initialized memory even if crypto_newsession has never
succeeded. So it was up to every driver in tree to gracefully handle
sid=0, if it happened to get assigned hid=0. And, as long as the
freesession callback was expected to just return an error code when
given a bogus session id, that worked out fine...because nothing ever
used the error code.
That was a terrible fragile system that should never have been
invented. Instead, let's just ensure that valid session ids are
nonzero, and make crypto_freesession with sid=0 be a no-op.
CRYPTO_F_USER is no longer needed. It was introduced in 2008 by
darran@ in crypto.c 1.30, cryptodev.c 1.45 in an attempt to avoid
double-free between the issuing thread and asynchronous callback.
But the `fix' didn't work. In 2017, knakahara@ fixed it properly in
cryptodev.c 1.87 by distinguishing `the crypto operation has
completed' (CRYPTO_F_DONE) from `the callback is done touching the
crp object' (CRYPTO_F_DQRETQ, now renamed to CRYPTODEV_F_RET).
CRYPTO_F_CBIMM formerly served to invoke the callback synchronously
from the driver's interrupt completion routine, to reduce contention
on what was once a single cryptoret thread. Now, there is a per-CPU
queue and softint for much cheaper processing, so there is less
motivation for this in the first place. So let's remove the
complicated logic. This means the callbacks never run in hard
interrupt context, which means we don't need to worry about recursion
into crypto_dispatch in hard interrupt context.
crypto_op() waited to be set CRYPTO_F_DONE with crp->crp_cv.
However, there is context switch chances between being set CRYPTO_F_DONE in
crypto_done() and done cv_signal(crp->crp_cv) in cryptodev_cb(), that is,
cryptodev_op() thread can run to cv_destroy(crp->crp_cv) before cryptoret()
thread is waken up. As a result, cryptodev_cb() can call invalid(destroyed)
cv_signal(crp->crp_cv).
Furthermore, below two implementations cause other races.
- waiting CRYPTO_F_DONE with crp->crp_cv
- context witch chances between set CRYPTO_F_DONE and cv_signal(crp->crp_cv)
So, use other flag(CRYPTO_F_DQRETQ) for cryptodev_op() and cryptodev_key(),
and then call cv_signal(crp->crp_cv) immediately after set CRYPTO_F_DQRETQ.
Tested concurrent over 20 processes with software and hardware drivers.
and can not disappear -- no need to hold crypto_mtx to check the
driver list
(the whole check is questionable)
-crp->crp_cv (the condition variable) is used by userland cryptodev
exclusively -- move its initialization there, no need to waste
cycles of in-kernel callers
-add a comment which members of "struct cryptop" are used
by opencrypto(9) and which by crypto(4)
(this should be split, no need to waste memory for in-kernel callers)
This is still somewhat experimental. Tested between 2 similar boxes
so far. There is much potential for performance improvement. For now,
I've changed the gmac code to accept any data alignment, as the "char *"
pointer suggests. As the code is practically used, 32-bit alignment
can be assumed, at the cost of data copies. I don't know whether
bytewise access or copies are worse performance-wise. For efficient
implementations using SSE2 instructions on x86, even stricter
alignment requirements might arise.
For this to fit, an API change in cryptosoft was adopted from OpenBSD
(addition of a "Setkey" method to hashes) which was done for GCM/GMAC
support there, so it might be useful in the future anyway.
tested against KAME IPSEC
AFAICT, FAST_IPSEC now supports as much as KAME.
-RFC2104 says that the block size of the hash algorithm must be used
for key/ipad/opad calculations. While formerly all ciphers used a block
length of 64, SHA384 and SHA512 use 128 bytes. So we can't use the
HMAC_BLOCK_LEN constant anymore. Add a new field to "struct auth_hash"
for the per-cipher blocksize.
-Due to this, there can't be a single "CRYPTO_SHA2_HMAC" external name
anymore. Replace this by 3 for the 3 different keysizes.
This was done by Open/FreeBSD before.
-Also fix the number of authenticator bits used tor ESP and AH to
conform to RFC4868, and remove uses of AH_HMAC_HASHLEN which did
assume a fixed authenticator size of 12 bytes.
FAST_IPSEC will not interoperate with KAME IPSEC anymore if sha2 is used,
because the latter doesn't implement these standards. It should
interoperate with at least modern Free/OpenBSD now.
(I've only tested with NetBSD-current/FAST_IPSEC on both ends.)
decompression:
-seperate the IPCOMP specific rule that compression must not grow the
data from general compression semantics: Introduce a special name
CRYPTO_DEFLATE_COMP_NOGROW/comp_algo_deflate_nogrow to describe
the IPCOMP semantics and use it there. (being here, fix the check
so that equal size is considered failure as well as required by
RFC2393)
Customers of CRYPTO_DEFLATE_COMP/comp_algo_deflate now always get
deflated data back, even if they are not smaller than the original.
-allow to pass a "size hint" to the DEFLATE decompression function
which is used for the initial buffer allocation. Due to the changes
done there, additional allocations and extra copies are avoided if the
initial allocation is sufficient. Set the size hint to MCLBYTES (=2k)
in IPCOMP which should be good for many use cases.
Extends the Opencrypto API to allow the destination buffer size to be
specified when its not the same size as the input buffer (i.e. for
operations like compress and decompress).
The crypto_op and crypt_n_op structures gain a u_int dst_len field.
The session_op structure gains a comp_alg field to specify a compression
algorithm.
Moved four ioctls to new ids; CIOCGSESSION, CIOCNGSESSION, CIOCCRYPT,
and CIOCNCRYPTM.
Added four backward compatible ioctls; OCIOCGSESSION, OCIOCNGSESSION,
OCIOCCRYPT, and OCIOCNCRYPTM.
Backward compatibility is maintained in ocryptodev.h and ocryptodev.c which
implement the original ioctls and set dst_len and comp_alg to 0.
Adds user-space access to compression features.
Adds software gzip support (CRYPTO_GZIP_COMP).
Adds the fast version of crc32 from zlib to libkern. This should be generally
useful and provide a place to start normalizing the various crc32 routines
in the kernel. The crc32 routine is used in this patch to support GZIP.
With input and support from tls@NetBSD.org.
completed by the crypto device, queued on the retq, but freed by the
ioctl lwp. The problem manifests as various panics relating to the
condvar inside the request. The problem can occur whenever the crypto
device completes the request immediately and the ioctl skips the cv_wait().
The problem can be reproduced by enabling cryptosoft and running an openssl
speed test. E.g.
sysctl -w kern.cryptodevallowsoft=-1
openssl speed -engine cryptodev -evp des-ede3-cbc -multi 64
Add a macro for TAILQ_FOREACH_REVERSE_SAFE() to queue.h, since this
was missing and the opencrypto code removes requests from a list while
iterating with TAILQ_FOREACH_REVERSE().
Add missing cv_destroy() calls for the key request cleanup.
Reviewed by Thor Lancelot Simon.
* Asynchronous operation with result retrieval via select/poll
* Mutliple-request submit/retrieve ioctls
* Mutliple-session create-destroy ioctls
Revise/rewrite crypto.4 manual page. It should now be much easier to write
new applications to this API.
Measured performance for trivial requests: 84,000 very short modular math
operations/sec, 120,000 very short md5 hashes per sec (with a hardware
accellerator of moderate performance but very low latency, whose driver
will be contributed at a later date).
Contributed to TNF by Coyote Point Systems, Inc.
(actually splnet) and condvars instead of tsleep/wakeup. Fix a few
miscellaneous problems and add some debugging printfs while there.
Restore set of CRYPTO_F_DONE in crypto_done() which was lost at some
point after this code came from FreeBSD -- it made it impossible to wait
properly for a condition.
Add flags analogous to the "crp" flags to the key operation's krp struct.
Add a new flag, CRYPTO_F_ONRETQ which tells us a request finished before
the kthread had a chance to dequeue it and call its callback -- this was
letting requests stick on the queues before even though done and copied
out.
Callers of crypto_newsession() or crypto_freesession() must now take the
mutex. Change netipsec to do so. Dispatch takes the mutex itself as
needed.
This was tested fairly extensively with the cryptosoft backend and lightly
with a new hardware driver. It has not been tested with FAST_IPSEC; I am
unable to ascertain whether FAST_IPSEC currently works at all in our tree.
pjd@FreeBSD.ORG, ad@NetBSD.ORG, and darran@snark.us pointed me in the
right direction several times in the course of this. Remaining bugs
are mine alone.
been in in our tree, and certainly does not work on any version of FreeBSD
now. Run through unifdef -D__NetBSD__ -U__FreeBSD__ yielding a small
reduction of size and a dramatic improvement in readability.
No, this does not yield any meaningful decrease in patchability (unlike
mechanical changes that touch live source lines) -- try it and see.
framework. There is no need to waste the space if you are only using
algoritms provided by hardware accelerators. To get the software
implementations, add "pseudo-device swcr" to your kernel config.
- Lazily initialize the opencrypto framework when crypto drivers
(either hardware or swcr) register themselves with the framework.
Sam Leffler's FreeBSD commit message was
``to eliminate context switch when returning results from the
software crypto driver''
but the patch also contains the CRYPTO_SESID*() macros used in newer
ubsec and hifn drivers.
pseudo-device to init_main(), so the framework is ready for
registration requests at autoconfiguration time.
Thanks to Quentin Garnier for confirming the change was required, and
for testing a similar fix.
riastradh