On AEAD decryption authentication failure we are suppose to
zero out the output plaintext buffer. However, we've missed
skipping the optional associated data that may prefix the
ciphertext. This commit fixes this issue.
Signed-off-by: NGilad Ben-Yossef <gilad@benyossef.com>
Fixes: e88b27c8 ("crypto: ccree - use std api sg_zero_buffer")
Cc: stable@vger.kernel.org
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Fixed a typo in a commnet.
Signed-off-by: NHadar Gat <hadar.gat@arm.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Fixed typos in ccree error msgs.
Signed-off-by: NHadar Gat <hadar.gat@arm.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

These drivers no longer need it as they are only probed via DT.
crypto_platform_data was allocated but unused, so remove it.
This is a follow up for:
commit 45a536e3 ("crypto: atmel-tdes - Retire dma_request_slave_channel_compat()")
commit db28512f ("crypto: atmel-sha - Retire dma_request_slave_channel_compat()")
commit 62f72cbd ("crypto: atmel-aes - Retire dma_request_slave_channel_compat()")
Signed-off-by: NTudor Ambarus <tudor.ambarus@microchip.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

In order to avoid CFI function prototype mismatches, this removes the
casts on assembly implementations of sha1/256/512 accelerators. The
safety checks from BUILD_BUG_ON() remain.

Additionally, this renames various arguments for clarity, as suggested
by Eric Biggers.
Signed-off-by: NKees Cook <keescook@chromium.org>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Do not return error when max stid reached, to Fallback to nic mode.
Signed-off-by: NVinay Kumar Yadav <vinay.yadav@chelsio.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

corrected function call context and moved t4_defer_reply
to apropriate location.
Signed-off-by: NVinay Kumar Yadav <vinay.yadav@chelsio.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Fixes: 8d818c10 ("crypto: caam/qi2 - add DPAA2-CAAM driver")
Signed-off-by: NHoria Geantă <horia.geanta@nxp.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

"AEAD" is capitalized everywhere else.
Use "an" when followed by a written or spoken vowel.

Fixes: be1eb7f7 ("crypto: essiv - create wrapper template for ESSIV generation")
Signed-off-by: NGeert Uytterhoeven <geert@linux-m68k.org>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

This branch prediction macro on the hot path can improve
small performance(about 2%) according to the test.
Signed-off-by: NZaibo Xu <xuzaibo@huawei.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Reorder the input parameters of hpre_crt_para_get to
make it cleaner.
Signed-off-by: NZaibo Xu <xuzaibo@huawei.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

1.Use memzero_explicit to clear key;
2.Fix some little endian writings;
3.Fix some other bugs and stuff of code style;
Signed-off-by: NZaibo Xu <xuzaibo@huawei.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

1.Fixed the bug of software tfm leakage.
2.Update HW error log message.
Signed-off-by: NZaibo Xu <xuzaibo@huawei.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

authenc(hmac(sha1),cbc(aes)), authenc(hmac(sha256),cbc(aes)), and
authenc(hmac(sha512),cbc(aes)) support are added for SEC v2.
Signed-off-by: NZaibo Xu <xuzaibo@huawei.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

1.Define base initiation of QP for context which can be reused.
2.Define cipher initiation for other algorithms.
Signed-off-by: NZaibo Xu <xuzaibo@huawei.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

After adding branch prediction for skcipher hot path,
a little bit income of performance is gotten.
Signed-off-by: NZaibo Xu <xuzaibo@huawei.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Add error type parameter for call back checking inside.
Signed-off-by: NZaibo Xu <xuzaibo@huawei.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

1.Adjust call back function.
2.Adjust parameter checking function.
Signed-off-by: NZaibo Xu <xuzaibo@huawei.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

1.Put resource including request and resource list into
  QP context structure to avoid allocate memory repeatedly.
2.Add max context queue number to void kcalloc large memory for QP context.
3.Remove the resource allocation operation.
4.Redefine resource allocation APIs to be shared by other algorithms.
5.Move resource allocation and free inner functions out of
  operations 'struct sec_req_op', and they are called directly.
Signed-off-by: NZaibo Xu <xuzaibo@huawei.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

1.Adjust dma map function to be reused by AEAD algorithms;
2.Update some names of internal functions and variables to
  support AEAD algorithms;
3.Rename 'sec_skcipher_exit' as 'sec_skcipher_uninit';
4.Rename 'sec_get/put_queue_id' as 'sec_alloc/free_queue_id';
Signed-off-by: NZaibo Xu <xuzaibo@huawei.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Fixed some print, coding style and comments of HiSilicon SEC V2.
Signed-off-by: NZaibo Xu <xuzaibo@huawei.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Applied some advices of Marco Elver on atomic usage of Debugfs,
which is carried out by basing on Arnd Bergmann's fixing patch.
Reported-by: NArnd Bergmann <arnd@arndb.de>
Reported-by: NMarco Elver <elver@google.com>
Signed-off-by: NZaibo Xu <xuzaibo@huawei.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Remove NULL check for pool variable, since in the current
code path it is guaranteed to be non-NULL.
Reported-by: NDan Carpenter <dan.carpenter@oracle.com>
Signed-off-by: NRijo Thomas <Rijo-john.Thomas@amd.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Rename err label to err_device_unregister for better
readability.
Suggested-by: NDan Carpenter <dan.carpenter@oracle.com>
Signed-off-by: NRijo Thomas <Rijo-john.Thomas@amd.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Currently, if tee_device_alloc() fails, then tee_device_unregister()
is a no-op. Therefore, skip the function call to tee_device_unregister() by
introducing a new goto label 'err_free_pool'.
Reported-by: NDan Carpenter <dan.carpenter@oracle.com>
Signed-off-by: NRijo Thomas <Rijo-john.Thomas@amd.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

If there is no TEE with which the driver can communicate, then
print an error message and return.
Suggested-by: NDan Carpenter <dan.carpenter@oracle.com>
Signed-off-by: NRijo Thomas <Rijo-john.Thomas@amd.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Remove unused variable initialization from driver code.

If enabled as a compiler option, compiler may throw warning for
unused assignments.
Reported-by: NDan Carpenter <dan.carpenter@oracle.com>
Fixes: 757cc3e9 ("tee: add AMD-TEE driver")
Signed-off-by: NRijo Thomas <Rijo-john.Thomas@amd.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

When the kernel XTS implementation was extended to deal with ciphertext
stealing in commit 8083b1bf ("crypto: xts - add support for ciphertext
stealing"), a check was added to reject inputs that were too short.

However, in the vmx enablement - commit 23966841 ("crypto: vmx/xts -
use fallback for ciphertext stealing"), that check wasn't added to the
vmx implementation. This disparity leads to errors like the following:

alg: skcipher: p8_aes_xts encryption unexpectedly succeeded on test vector "random: len=0 klen=64"; expected_error=-22, cfg="random: inplace may_sleep use_finup src_divs=[<flush>66.99%@+10, 33.1%@alignmask+1155]"

Return -EINVAL if asked to operate with a cryptlen smaller than the AES
block size. This brings vmx in line with the generic implementation.
Reported-by: NErhard Furtner <erhard_f@mailbox.org>
Link: https://bugzilla.kernel.org/show_bug.cgi?id=206049
Fixes: 23966841 ("crypto: vmx/xts - use fallback for ciphertext stealing")
Cc: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Cc: stable@vger.kernel.org # v5.4+
Signed-off-by: NMichael Ellerman <mpe@ellerman.id.au>
[dja: commit message]
Signed-off-by: NDaniel Axtens <dja@axtens.net>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

If CRYPTO_CURVE25519 is y, CRYPTO_LIB_CURVE25519_GENERIC will be
y, but CRYPTO_LIB_CURVE25519 may be set to m, this causes build
errors:

lib/crypto/curve25519-selftest.o: In function `curve25519':
curve25519-selftest.c:(.text.unlikely+0xc): undefined reference to `curve25519_arch'
lib/crypto/curve25519-selftest.o: In function `curve25519_selftest':
curve25519-selftest.c:(.init.text+0x17e): undefined reference to `curve25519_base_arch'

This is because the curve25519 self-test code is being controlled
by the GENERIC option rather than the overall CURVE25519 option,
as is the case with blake2s.  To recap, the GENERIC and ARCH options
for CURVE25519 are internal only and selected by users such as
the Crypto API, or the externally visible CURVE25519 option which
in turn is selected by wireguard.  The self-test is specific to the
the external CURVE25519 option and should not be enabled by the
Crypto API.

This patch fixes this by splitting the GENERIC module from the
CURVE25519 module with the latter now containing just the self-test.
Reported-by: NHulk Robot <hulkci@huawei.com>
Fixes: aa127963 ("crypto: lib/curve25519 - re-add selftests")
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>
Reviewed-by: NJason A. Donenfeld <Jason@zx2c4.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Add support for the crypto engine used in i.mx8mn (i.MX 8M "Nano"),
which is very similar to the one used in i.mx8mq, i.mx8mm.

Since the clocks are identical for all members of i.MX 8M family,
simplify the SoC <--> clock array mapping table.
Signed-off-by: NHoria Geantă <horia.geanta@nxp.com>
Tested-by: NIuliana Prodan <iuliana.prodan@nxp.com>
Reviewed-by: NIuliana Prodan <iuliana.prodan@nxp.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Some code were left in the final driver but without any use.
Signed-off-by: NCorentin Labbe <clabbe.montjoie@gmail.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Removing the driver cause an oops due to the fact we clean an extra
channel.
Let's give the right index to the cleaning function.

Fixes: 06f751b6 ("crypto: allwinner - Add sun8i-ce Crypto Engine")
Signed-off-by: NCorentin Labbe <clabbe.montjoie@gmail.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Removing the driver cause an oops due to the fact we clean an extra
channel.
Let's give the right index to the cleaning function.
Fixes: 48fe583f ("crypto: amlogic - Add crypto accelerator for amlogic GXL")
Signed-off-by: NCorentin Labbe <clabbe.montjoie@gmail.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Removing the driver cause an oops due to the fact we clean an extra
channel.
Let's give the right index to the cleaning function.
Fixes: f08fcced ("crypto: allwinner - Add sun8i-ss cryptographic offloader")
Signed-off-by: NCorentin Labbe <clabbe.montjoie@gmail.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

This appears to be some kind of copy and paste error, and is actually
dead code.

Pre: f = 0 ⇒ (f >> 32) = 0
    f = (f >> 32) + le32_to_cpu(digest[0]);
Post: 0 ≤ f < 2³²
    put_unaligned_le32(f, dst);

Pre: 0 ≤ f < 2³² ⇒ (f >> 32) = 0
    f = (f >> 32) + le32_to_cpu(digest[1]);
Post: 0 ≤ f < 2³²
    put_unaligned_le32(f, dst + 4);

Pre: 0 ≤ f < 2³² ⇒ (f >> 32) = 0
    f = (f >> 32) + le32_to_cpu(digest[2]);
Post: 0 ≤ f < 2³²
    put_unaligned_le32(f, dst + 8);

Pre: 0 ≤ f < 2³² ⇒ (f >> 32) = 0
    f = (f >> 32) + le32_to_cpu(digest[3]);
Post: 0 ≤ f < 2³²
    put_unaligned_le32(f, dst + 12);

Therefore this sequence is redundant. And Andy's code appears to handle
misalignment acceptably.
Signed-off-by: NJason A. Donenfeld <Jason@zx2c4.com>
Tested-by: NArd Biesheuvel <ardb@kernel.org>
Reviewed-by: NArd Biesheuvel <ardb@kernel.org>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

These x86_64 vectorized implementations support AVX, AVX-2, and AVX512F.
The AVX-512F implementation is disabled on Skylake, due to throttling,
but it is quite fast on >= Cannonlake.

On the left is cycle counts on a Core i7 6700HQ using the AVX-2
codepath, comparing this implementation ("new") to the implementation in
the current crypto api ("old"). On the right are benchmarks on a Xeon
Gold 5120 using the AVX-512 codepath. The new implementation is faster
on all benchmarks.

        AVX-2                  AVX-512
      ---------              -----------

    size    old     new      size   old     new
    ----    ----    ----     ----   ----    ----
    0       70      68       0      74      70
    16      92      90       16     96      92
    32      134     104      32     136     106
    48      172     120      48     184     124
    64      218     136      64     218     138
    80      254     158      80     260     160
    96      298     174      96     300     176
    112     342     192      112    342     194
    128     388     212      128    384     212
    144     428     228      144    420     226
    160     466     246      160    464     248
    176     510     264      176    504     264
    192     550     282      192    544     282
    208     594     302      208    582     300
    224     628     316      224    624     318
    240     676     334      240    662     338
    256     716     354      256    708     358
    272     764     374      272    748     372
    288     802     352      288    788     358
    304     420     366      304    422     370
    320     428     360      320    432     364
    336     484     378      336    486     380
    352     426     384      352    434     390
    368     478     400      368    480     408
    384     488     394      384    490     398
    400     542     408      400    542     412
    416     486     416      416    492     426
    432     534     430      432    538     436
    448     544     422      448    546     432
    464     600     438      464    600     448
    480     540     448      480    548     456
    496     594     464      496    594     476
    512     602     456      512    606     470
    528     656     476      528    656     480
    544     600     480      544    606     498
    560     650     494      560    652     512
    576     664     490      576    662     508
    592     714     508      592    716     522
    608     656     514      608    664     538
    624     708     532      624    710     552
    640     716     524      640    720     516
    656     770     536      656    772     526
    672     716     548      672    722     544
    688     770     562      688    768     556
    704     774     552      704    778     556
    720     826     568      720    832     568
    736     768     574      736    780     584
    752     822     592      752    826     600
    768     830     584      768    836     560
    784     884     602      784    888     572
    800     828     610      800    838     588
    816     884     628      816    884     604
    832     888     618      832    894     598
    848     942     632      848    946     612
    864     884     644      864    896     628
    880     936     660      880    942     644
    896     948     652      896    952     608
    912     1000    664      912    1004    616
    928     942     676      928    954     634
    944     994     690      944    1000    646
    960     1002    680      960    1008    646
    976     1054    694      976    1062    658
    992     1002    706      992    1012    674
    1008    1052    720      1008   1058    690

This commit wires in the prior implementation from Andy, and makes the
following changes to be suitable for kernel land.

  - Some cosmetic and structural changes, like renaming labels to
    .Lname, constants, and other Linux conventions, as well as making
    the code easy for us to maintain moving forward.

  - CPU feature checking is done in C by the glue code.

  - We avoid jumping into the middle of functions, to appease objtool,
    and instead parameterize shared code.

  - We maintain frame pointers so that stack traces make sense.

  - We remove the dependency on the perl xlate code, which transforms
    the output into things that assemblers we don't care about use.

Importantly, none of our changes affect the arithmetic or core code, but
just involve the differing environment of kernel space.
Signed-off-by: NJason A. Donenfeld <Jason@zx2c4.com>
Signed-off-by: NSamuel Neves <sneves@dei.uc.pt>
Co-developed-by: NSamuel Neves <sneves@dei.uc.pt>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

These x86_64 vectorized implementations come from Andy Polyakov's
CRYPTOGAMS implementation, and are included here in raw form without
modification, so that subsequent commits that fix these up for the
kernel can see how it has changed.
Signed-off-by: NJason A. Donenfeld <Jason@zx2c4.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

These two C implementations from Zinc -- a 32x32 one and a 64x64 one,
depending on the platform -- come from Andrew Moon's public domain
poly1305-donna portable code, modified for usage in the kernel. The
precomputation in the 32-bit version and the use of 64x64 multiplies in
the 64-bit version make these perform better than the code it replaces.
Moon's code is also very widespread and has received many eyeballs of
scrutiny.

There's a bit of interference between the x86 implementation, which
relies on internal details of the old scalar implementation. In the next
commit, the x86 implementation will be replaced with a faster one that
doesn't rely on this, so none of this matters much. But for now, to keep
this passing the tests, we inline the bits of the old implementation
that the x86 implementation relied on. Also, since we now support a
slightly larger key space, via the union, some offsets had to be fixed
up.

Nonce calculation was folded in with the emit function, to take
advantage of 64x64 arithmetic. However, Adiantum appeared to rely on no
nonce handling in emit, so this path was conditionalized. We also
introduced a new struct, poly1305_core_key, to represent the precise
amount of space that particular implementation uses.

Testing with kbench9000, depending on the CPU, the update function for
the 32x32 version has been improved by 4%-7%, and for the 64x64 by
19%-30%. The 32x32 gains are small, but I think there's great value in
having a parallel implementation to the 64x64 one so that the two can be
compared side-by-side as nice stand-alone units.
Signed-off-by: NJason A. Donenfeld <Jason@zx2c4.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Merge crypto tree to pick up hisilicon patch.

Fix up inconsistent usage of upper and lowercase letters in "Exynos"
name.

"EXYNOS" is not an abbreviation but a regular trademarked name.
Therefore it should be written with lowercase letters starting with
capital letter.

The lowercase "Exynos" name is promoted by its manufacturer Samsung
Electronics Co., Ltd., in advertisement materials and on website.
Signed-off-by: NKrzysztof Kozlowski <krzk@kernel.org>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>