The CCP has the ability to perform several operations simultaneously,
but only one interrupt.  When implemented as a PCI device and using
MSI-X/MSI interrupts, use a tasklet model to service interrupts. By
disabling and enabling interrupts from the CCP, coupled with the
queuing that tasklets provide, we can ensure that all events
(occurring on the device) are recognized and serviced.

This change fixes a problem wherein 2 or more busy queues can cause
notification bits to change state while a (CCP) interrupt is being
serviced, but after the queue state has been evaluated. This results
in the event being 'lost' and the queue hanging, waiting to be
serviced. Since the status bits are never fully de-asserted, the
CCP never generates another interrupt (all bits zero -> one or more
bits one), and no further CCP operations will be executed.

Cc: <stable@vger.kernel.org> # 4.9.x+
Signed-off-by: NGary R Hook <gary.hook@amd.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

The CCP has the ability to perform several operations simultaneously,
but only one interrupt.  When implemented as a PCI device and using
MSI-X/MSI interrupts, use a tasklet model to service interrupts. By
disabling and enabling interrupts from the CCP, coupled with the
queuing that tasklets provide, we can ensure that all events
(occurring on the device) are recognized and serviced.

This change fixes a problem wherein 2 or more busy queues can cause
notification bits to change state while a (CCP) interrupt is being
serviced, but after the queue state has been evaluated. This results
in the event being 'lost' and the queue hanging, waiting to be
serviced. Since the status bits are never fully de-asserted, the
CCP never generates another interrupt (all bits zero -> one or more
bits one), and no further CCP operations will be executed.

Cc: <stable@vger.kernel.org> # 4.9.x+
Signed-off-by: NGary R Hook <gary.hook@amd.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

trivial spelling mistake, missing r, rename to ce_ring_control
Signed-off-by: NColin Ian King <colin.king@canonical.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

The cipher_null is not a real cipher, FIPS mode should not restrict its use.

It is used for several tests (for example in cryptsetup testsuite) and also
temporarily for reencryption of not yet encrypted device in cryptsetup-reencrypt tool.

Problem is easily reproducible with
  cryptsetup benchmark -c null
Signed-off-by: NMilan Broz <gmazyland@gmail.com>
Acked-by: NStephan Müller <smueller@chronox.de>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

This reverts commit 42ae2922.  It
causes a regression with older versions of gcc.  The consensus is
that this should instead be fixed in clang.
Reported-by: NPaul Gortmaker <paul.gortmaker@windriver.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Ensure that we disable interrupts first when shutting down
the driver.

Cc: <stable@vger.kernel.org> # 4.9.x+
Signed-off-by: NGary R Hook <ghook@amd.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Each CCP queue can product interrupts for 4 conditions:
operation complete, queue empty, error, and queue stopped.
This driver only works with completion and error events.

Cc: <stable@vger.kernel.org> # 4.9.x+
Signed-off-by: NGary R Hook <gary.hook@amd.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

This patch adds support for hardware random generator on MT7623 SoC
and should also work on other similar Mediatek SoCs. Currently,
the driver is already tested successfully with rng-tools.
Signed-off-by: NSean Wang <sean.wang@mediatek.com>
Reviewed-by: NPrasannaKumar Muralidharan <prasannatsmkumar@gmail.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Document the devicetree bindings for Mediatek random number
generator which could be found on MT7623 SoC or other similar
Mediatek SoCs.
Signed-off-by: NSean Wang <sean.wang@mediatek.com>
Acked-by: NRob Herring <robh@kernel.org>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

In crct10dif_vpmsum() we call enable_kernel_altivec() without first
disabling preemption, which is not allowed.

It used to be sufficient just to call pagefault_disable(), because that
also disabled preemption. But the two were decoupled in commit 8222dbe2
("sched/preempt, mm/fault: Decouple preemption from the page fault
logic") in mid 2015.

The crct10dif-vpmsum code inherited this bug from the crc32c-vpmsum code
on which it was modelled.

So add the missing preempt_disable/enable(). We should also call
disable_kernel_fp(), although it does nothing by default, there is a
debug switch to make it active and all enables should be paired with
disables.

Fixes: b01df1c1 ("crypto: powerpc - Add CRC-T10DIF acceleration")
Acked-by: NDaniel Axtens <dja@axtens.net>
Signed-off-by: NMichael Ellerman <mpe@ellerman.id.au>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Compression implementations might return valid outputs that
do not match what specified in the test vectors.
For this reason, the testmgr might report that a compression
implementation failed the test even if the data produced
by the compressor is correct.
This implements a decompress-and-verify test for acomp
compression tests rather than a known answer test.
Signed-off-by: NGiovanni Cabiddu <giovanni.cabiddu@intel.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Add crypto_register_acomps and crypto_unregister_acomps to allow
the registration of multiple implementations with one call.
Signed-off-by: NGiovanni Cabiddu <giovanni.cabiddu@intel.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

* A multiplication for the size determination of a memory allocation
  indicated that an array data structure should be processed.
  Thus use the corresponding function "devm_kcalloc".

* Replace the specification of a data structure by a pointer dereference
  to make the corresponding size determination a bit safer according to
  the Linux coding style convention.
Signed-off-by: NMarkus Elfring <elfring@users.sourceforge.net>
Acked-by: NShannon Nelson <shannon.nelson@oracle.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Up to now, 'crypto_alloc_shash()' may return a valid pointer, an error
pointer or NULL (in case of invalid parameter)
Update it to always return an error pointer in case of error. It now
returns ERR_PTR(-EINVAL) instead of NULL in case of invalid parameter.

This simplifies error handling.

Also fix a crash in 'chcr_authenc_setkey()' if 'chcr_alloc_shash()'
returns an error pointer and the "goto out" path is taken.
Signed-off-by: NChristophe JAILLET <christophe.jaillet@wanadoo.fr>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Per Dan's static checker warning, the code that returns NULL was removed
in 2010, so this patch updates the comments and fixes the code
assumptions.
Signed-off-by: NJason A. Donenfeld <Jason@zx2c4.com>
Reported-by: NDan Carpenter <dan.carpenter@oracle.com>
Acked-by: NSteffen Klassert <steffen.klassert@secunet.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Replace existing hw_ranndom/exynos-rng driver with a new, reworked one.
This is a driver for pseudo random number generator block which on
Exynos4 chipsets must be seeded with some value.  On newer Exynos5420
chipsets it might seed itself from true random number generator block
but this is not implemented yet.

New driver is a complete rework to use the crypto ALGAPI instead of
hw_random API.  Rationale for the change:
1. hw_random interface is for true RNG devices.
2. The old driver was seeding itself with jiffies which is not a
   reliable source for randomness.
3. Device generates five random 32-bit numbers in each pass but old
   driver was returning only one 32-bit number thus its performance was
   reduced.

Compatibility with DeviceTree bindings is preserved.

New driver does not use runtime power management but manually enables
and disables the clock when needed.  This is preferred approach because
using runtime PM just to toggle clock is huge overhead.

Another difference is reseeding itself with generated random data
periodically and during resuming from system suspend (previously driver
was re-seeding itself again with jiffies).
Signed-off-by: NKrzysztof Kozlowski <krzk@kernel.org>
Reviewed-by: NStephan Müller <smueller@chronox.de>
Reviewed-by: NPrasannaKumar Muralidharan <prasannatsmkumar@gmail.com>
Reviewed-by: NBartlomiej Zolnierkiewicz <b.zolnierkie@samsung.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Few parts of kernel define their own macro for aligning down so provide
a common define for this, with the same usage and assumptions as existing
ALIGN.

Convert also three existing implementations to this one.
Signed-off-by: NKrzysztof Kozlowski <krzk@kernel.org>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Fix to return error code -ENOMEM from the kmem_cache_create() error
handling case instead of 0(err is 0 here), as done elsewhere in this
function.

Fixes: 67c2315d ("crypto: caam - add Queue Interface (QI) backend support")
Signed-off-by: NWei Yongjun <weiyongjun1@huawei.com>
Acked-by: NHoria Geantă <horia.geanta@nxp.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Fallback to sw when
    I AAD length greater than 511
    II Zero length payload
    II No of sg entries exceeds Request size.
Signed-off-by: NHarsh Jain <harsh@chelsio.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

The patch fixes a critical issue to map txqid with flows on the hardware appropriately,
if tx queues created are more than flows configured then  txqid shall map within
the range of hardware flows configured. This ensure that un-mapped txqid does not remain un-handled.
The patch also segregated the rxqid and txqid for clarity.
Signed-off-by: NAtul Gupta <atul.gupta@chelsio.com>
Reviewed-by: NGanesh Goudar <ganeshgr@chelsio.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Use hmac_ctrl bit value saved in setauthsize callback.
Signed-off-by: NHarsh Jain <harsh@chelsio.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

templates(gcm,ccm etc) inherit priority value of driver to
calculate its priority. In some cases template priority becomes
 more than driver priority for same algo.
Without this patch we will not be able to use driver authenc algos. It will
be good if it pushed in stable kernel.
Signed-off-by: NHarsh Jain <harsh@chelsio.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Decompress function in LZ4 library is supposed to return an error code or
negative result. But, it returns -1 when any error is detected. Return
error code when the library returns negative value.
Signed-off-by: NMyungho Jung <mhjungk@gmail.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

The author meant to free the variable that was just allocated, instead
of the one that failed to be allocated, but made a simple typo. This
patch rectifies that.
Signed-off-by: NJason A. Donenfeld <Jason@zx2c4.com>
Cc: stable@vger.kernel.org
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

With the new explicit IV generators, we may now exceed the 64-byte
length limit on the algorithm name, e.g., with

	echainiv(authencesn(hmac(sha256-generic),cbc(des3_ede-generic)))

This patch extends the length limit to 128 bytes.
Reported-by: NAlexander Sverdlin <alexander.sverdlin@nokia.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>
Acked-by: NAlexander Sverdlin <alexander.sverdlin@nokia.com>
Tested-by: NAlexander Sverdlin <alexander.sverdlin@nokia.com>

This patch fixes the xfrm_user code to use the actual array size
rather than the hard-coded CRYPTO_MAX_ALG_NAME length.  This is
because the array size is fixed at 64 bytes while we want to increase
the in-kernel CRYPTO_MAX_ALG_NAME value.
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>
Acked-by: NAlexander Sverdlin <alexander.sverdlin@nokia.com>
Tested-by: NAlexander Sverdlin <alexander.sverdlin@nokia.com>
Acked-by: NSteffen Klassert <steffen.klassert@secunet.com>

This patch removes the hard-coded 64-byte limit on the length
of the algorithm name through bind(2).  The address length can
now exceed that.  The user-space structure remains unchanged.
In order to use a longer name simply extend the salg_name array
beyond its defined 64 bytes length.
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

This patch hard-codes CRYPTO_MAX_NAME in the user-space API to
64, which is the current value of CRYPTO_MAX_ALG_NAME.  This patch
also replaces all remaining occurences of CRYPTO_MAX_ALG_NAME
in the user-space API with CRYPTO_MAX_NAME.

This way the user-space API will not be modified when we raise
the value of CRYPTO_MAX_ALG_NAME.

Furthermore, the code has been updated to handle names longer than
the user-space API.  They will be truncated.
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>
Acked-by: NAlexander Sverdlin <alexander.sverdlin@nokia.com>
Tested-by: NAlexander Sverdlin <alexander.sverdlin@nokia.com>

Some hardware RNGs provide a single register for obtaining random data.
Instead of signaling when new data is available, the reader must wait a
fixed amount of time between reads for new data to be generated.
timeriomem_rng implements this scheme with the period specified in
platform data or device tree.  While the period is specified in
microseconds, the implementation used a standard timer which has a
minimum delay of 1 jiffie and caused a significant bottleneck for
devices that can update at 1us.  By switching to an hrtimer, 1us periods
now only delay at most 2us per read.
Signed-off-by: NRick Altherr <raltherr@google.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

No functional changes.
Signed-off-by: NRick Altherr <raltherr@google.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Preserves the existing behavior of only returning 32-bits per call.
Signed-off-by: NRick Altherr <raltherr@google.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

The operand is an integer constant, make the constness explicit by
adding the modifier. This is needed for clang to generate valid code
and also works with gcc.

Also change the constraint of the operand from 'I' ("Integer constant
that is valid as an immediate operand in an ADD instruction", AArch64)
to 'i' ("An immediate integer operand").

Based-on-patch-from: Greg Hackmann <ghackmann@google.com>
Signed-off-by: NGreg Hackmann <ghackmann@google.com>
Signed-off-by: NMatthias Kaehlcke <mka@chromium.org>
Reviewed-by: NArd Biesheuvel <ard.biesheuvel@linaro.org>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Since the gf128mul_x_ble function used by xts.c is now defined inline
in the header file, the XTS module no longer depends on gf128mul.
Therefore, the 'select CRYPTO_GF128MUL' line can be safely removed.
Signed-off-by: NOndrej Mosnacek <omosnacek@gmail.com>
Reviewd-by: NEric Biggers <ebiggers@google.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

The le128_gf128mul_x_ble function in glue_helper.h is now obsolete and
can be replaced with the gf128mul_x_ble function from gf128mul.h.
Signed-off-by: NOndrej Mosnacek <omosnacek@gmail.com>
Reviewd-by: NEric Biggers <ebiggers@google.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Currently, gf128mul_x_ble works with pointers to be128, even though it
actually interprets the words as little-endian. Consequently, it uses
cpu_to_le64/le64_to_cpu on fields of type __be64, which is incorrect.

This patch fixes that by changing the function to accept pointers to
le128 and updating all users accordingly.
Signed-off-by: NOndrej Mosnacek <omosnacek@gmail.com>
Reviewd-by: NEric Biggers <ebiggers@google.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

The gf128mul_x_ble function is currently defined in gf128mul.c, because
it depends on the gf128mul_table_be multiplication table.

However, since the function is very small and only uses two values from
the table, it is better for it to be defined as inline function in
gf128mul.h. That way, the function can be inlined by the compiler for
better performance.

For consistency, the other gf128mul_x_* functions are also moved to the
header file. In addition, the code is rewritten to be constant-time.

After this change, the speed of the generic 'xts(aes)' implementation
increased from ~225 MiB/s to ~235 MiB/s (measured using 'cryptsetup
benchmark -c aes-xts-plain64' on an Intel system with CRYPTO_AES_X86_64
and CRYPTO_AES_NI_INTEL disabled).
Signed-off-by: NOndrej Mosnacek <omosnacek@gmail.com>
Reviewd-by: NEric Biggers <ebiggers@google.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Signed-off-by: NHaren Myneni <haren@us.ibm.com>
Acked-by: NDan Streetman <ddstreet@ieee.org>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

The AES GCM function (in ccp-ops) requires a fair amount of
stack space, which elicits a complaint when KASAN is enabled.
Rearranging and packing a few structures eliminates the
warning.
Signed-off-by: NGary R Hook <gary.hook@amd.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Endianness is dealt with when the command descriptor is
copied into the command queue. Remove any occurrences of
cpu_to_le32() found elsewhere.
Signed-off-by: NGary R Hook <gary.hook@amd.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Add STM32 crypto support in stm32_defconfig file.
Signed-off-by: NFabien Dessenne <fabien.dessenne@st.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>