This removes all the boilerplate from the existing implementation,
and replaces it with calls into the base layer.
Signed-off-by: NArd Biesheuvel <ard.biesheuvel@linaro.org>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

This removes all the boilerplate from the existing implementation,
and replaces it with calls into the base layer.
Signed-off-by: NArd Biesheuvel <ard.biesheuvel@linaro.org>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

This removes all the boilerplate from the existing implementation,
and replaces it with calls into the base layer.
Signed-off-by: NArd Biesheuvel <ard.biesheuvel@linaro.org>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Add Andy Polyakov's optimized assembly and NEON implementations for
SHA-256/224.

The sha256-armv4.pl script for generating the assembly code is from
OpenSSL commit 51f8d095562f36cdaa6893597b5c609e943b0565.

Compared to sha256-generic these implementations have the following
tcrypt speed improvements on Motorola Nexus 6 (Snapdragon 805):

  bs    b/u      sha256-neon  sha256-asm
  16    16       x1.32        x1.19
  64    16       x1.27        x1.15
  64    64       x1.36        x1.20
  256   16       x1.22        x1.11
  256   64       x1.36        x1.19
  256   256      x1.59        x1.23
  1024  16       x1.21        x1.10
  1024  256      x1.65        x1.23
  1024  1024     x1.76        x1.25
  2048  16       x1.21        x1.10
  2048  256      x1.66        x1.23
  2048  1024     x1.78        x1.25
  2048  2048     x1.79        x1.25
  4096  16       x1.20        x1.09
  4096  256      x1.66        x1.23
  4096  1024     x1.79        x1.26
  4096  4096     x1.82        x1.26
  8192  16       x1.20        x1.09
  8192  256      x1.67        x1.23
  8192  1024     x1.80        x1.26
  8192  4096     x1.85        x1.28
  8192  8192     x1.85        x1.27

Where bs refers to block size and b/u to bytes per update.
Signed-off-by: NSami Tolvanen <samitolvanen@google.com>
Cc: Andy Polyakov <appro@openssl.org>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Flag all Multi buffer SHA1 helper ciphers as internal ciphers
to prevent them from being called by normal users.
Signed-off-by: NStephan Mueller <smueller@chronox.de>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Flag all 64 bit ARMv8 AES helper ciphers as internal ciphers to
prevent them from being called by normal users.
Signed-off-by: NStephan Mueller <smueller@chronox.de>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Flag all ARMv8 AES helper ciphers as internal ciphers to prevent
them from being called by normal users.
Signed-off-by: NStephan Mueller <smueller@chronox.de>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Flag all NEON bit sliced AES helper ciphers as internal ciphers to
prevent them from being called by normal users.
Signed-off-by: NStephan Mueller <smueller@chronox.de>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Flag all Twofish AVX helper ciphers as internal ciphers to prevent
them from being called by normal users.
Signed-off-by: NStephan Mueller <smueller@chronox.de>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Flag all Serpent SSE2 helper ciphers as internal ciphers to prevent
them from being called by normal users.
Signed-off-by: NStephan Mueller <smueller@chronox.de>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Flag all Serpent AVX helper ciphers as internal ciphers to prevent
them from being called by normal users.
Signed-off-by: NStephan Mueller <smueller@chronox.de>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Flag all Serpent AVX2 helper ciphers as internal ciphers to prevent
them from being called by normal users.
Signed-off-by: NStephan Mueller <smueller@chronox.de>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Flag all CAST6 helper ciphers as internal ciphers to prevent them
from being called by normal users.
Signed-off-by: NStephan Mueller <smueller@chronox.de>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Flag all AVX Camellia helper ciphers as internal ciphers to prevent
them from being called by normal users.
Signed-off-by: NStephan Mueller <smueller@chronox.de>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Flag all CAST5 helper ciphers as internal ciphers to prevent them
from being called by normal users.
Signed-off-by: NStephan Mueller <smueller@chronox.de>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Flag all AES-NI Camellia helper ciphers as internal ciphers to
prevent them from being called by normal users.
Signed-off-by: NStephan Mueller <smueller@chronox.de>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Flag all GHASH ARMv8 vmull.p64 helper ciphers as internal ciphers
to prevent them from being called by normal users.
Signed-off-by: NStephan Mueller <smueller@chronox.de>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Flag all ash clmulni helper ciphers as internal ciphers to prevent them
from being called by normal users.
Signed-off-by: NStephan Mueller <smueller@chronox.de>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Flag all AES-NI helper ciphers as internal ciphers to prevent them from
being called by normal users.
Signed-off-by: NStephan Mueller <smueller@chronox.de>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

This fixes a bug in the new v8 Crypto Extensions GHASH code
that only manifests itself in big-endian mode.
Signed-off-by: NArd Biesheuvel <ard.biesheuvel@linaro.org>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

fixing a syntax-error .
Signed-off-by: NAmeen Ali <AmeenAli023@gmail.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

The semantic patch that fixes this problem is as follows:
(http://coccinelle.lip6.fr/)

// <smpl>
@r@
type T;
identifier f;
@@

static T f (...) { ... }

@@
identifier r.f;
declarer name EXPORT_SYMBOL_GPL;
@@

-EXPORT_SYMBOL_GPL(f);
// </smpl>
Signed-off-by: NJulia Lawall <Julia.Lawall@lip6.fr>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

This implements the GHASH hash algorithm (as used by the GCM AEAD
chaining mode) using the AArch32 version of the 64x64 to 128 bit
polynomial multiplication instruction (vmull.p64) that is part of
the ARMv8 Crypto Extensions.
Signed-off-by: NArd Biesheuvel <ard.biesheuvel@linaro.org>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

This implements the ECB, CBC, CTR and XTS asynchronous block ciphers
using the AArch32 versions of the ARMv8 Crypto Extensions for AES.
Signed-off-by: NArd Biesheuvel <ard.biesheuvel@linaro.org>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

This implements the SHA-224/256 secure hash algorithm using the AArch32
versions of the ARMv8 Crypto Extensions for SHA2.
Signed-off-by: NArd Biesheuvel <ard.biesheuvel@linaro.org>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

This implements the SHA1 secure hash algorithm using the AArch32
versions of the ARMv8 Crypto Extensions for SHA1.
Signed-off-by: NArd Biesheuvel <ard.biesheuvel@linaro.org>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

This moves all Kconfig symbols defined in crypto/Kconfig that depend
on CONFIG_ARM to a dedicated Kconfig file in arch/arm/crypto, which is
where the code that implements those features resides as well.
Signed-off-by: NArd Biesheuvel <ard.biesheuvel@linaro.org>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Add OCTEON SHA512 module.
Signed-off-by: NAaro Koskinen <aaro.koskinen@iki.fi>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Add OCTEON SHA256 module.
Signed-off-by: NAaro Koskinen <aaro.koskinen@iki.fi>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Add OCTEON SHA1 module.
Signed-off-by: NAaro Koskinen <aaro.koskinen@iki.fi>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Add instruction definitions for SHA1/256/512.
Signed-off-by: NAaro Koskinen <aaro.koskinen@iki.fi>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Always disable preemption on behalf of the drivers when crypto engine
is taken into use. This will simplify the usage.
Signed-off-by: NAaro Koskinen <aaro.koskinen@iki.fi>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Don't disable bottom half while the crypto engine is in use, as it
should be unnecessary: All kernel crypto engine usage is wrapped with
crypto engine state save/restore, so if we get interrupted by softirq
that uses crypto they should save and restore our context.

This actually fixes an issue when running OCTEON MD5 with interrupts
disabled (tcrypt mode=302). There's a WARNING because the module is
trying to enable the bottom half with irqs disabled:

[   52.656610] ------------[ cut here ]------------
[   52.661439] WARNING: CPU: 1 PID: 428 at /home/aaro/git/linux/kernel/softirq.c:150 __local_bh_enable_ip+0x9c/0xd8()
[   52.671780] Modules linked in: tcrypt(+)
[...]
[   52.763539] [<ffffffff8114082c>] warn_slowpath_common+0x94/0xd8
[   52.769465] [<ffffffff81144614>] __local_bh_enable_ip+0x9c/0xd8
[   52.775390] [<ffffffff81119574>] octeon_md5_final+0x12c/0x1e8
[   52.781144] [<ffffffff81337050>] shash_compat_digest+0xd0/0x1b0
Signed-off-by: NAaro Koskinen <aaro.koskinen@iki.fi>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

The current cryptodev-2.6 tree commits:

d9850fc5 ("crypto: powerpc/sha1 - kernel config")
50ba29aa ("crypto: powerpc/sha1 - glue")

failed to properly place files under arch/powerpc/crypto, which
leads to build errors:

make[1]: *** No rule to make target 'arch/powerpc/crypto/sha1-spe-asm.o', needed by 'arch/powerpc/crypto/sha1-ppc-spe.o'.  Stop.
make[1]: *** No rule to make target 'arch/powerpc/crypto/sha1_spe_glue.o', needed by 'arch/powerpc/crypto/sha1-ppc-spe.o'.  Stop.
Makefile:947: recipe for target 'arch/powerpc/crypto' failed

Move the two sha1 spe files under crypto/, and whilst there, rename
other powerpc crypto files with underscores to use dashes for
consistency.

Cc: Markus Stockhausen <stockhausen@collogia.de>
Signed-off-by: NKim Phillips <kim.phillips@freescale.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Integrate the module into the kernel config tree.
Signed-off-by: NMarkus Stockhausen <stockhausen@collogia.de>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Glue code for crypto infrastructure. Call the assembler
code where required. Take a little care about small input
data. Kick out early for input chunks < 64 bytes and replace
memset for context cleanup with simple loop.
Signed-off-by: NMarkus Stockhausen <stockhausen@collogia.de>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

This is the assembler code for the MD5 implementation.
Handling of algorithm constants has been slightly
changed to reduce register usage and make better use
of cores with multiple ALUs. Thus they are stored as
delta values.
Signed-off-by: NMarkus Stockhausen <stockhausen@collogia.de>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Integrate the module into the kernel config tree.
Signed-off-by: NMarkus Stockhausen <stockhausen@collogia.de>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Glue code for crypto infrastructure. Call the assembler
code where required. Disable preemption during calculation
and enable SPE instructions in the kernel prior to the
call. Avoid to disable preemption for too long.

Take a little care about small input data. Kick out early
for input chunks < 64 bytes and replace memset for context
cleanup with simple loop.
Signed-off-by: NMarkus Stockhausen <stockhausen@collogia.de>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

This is the assembler code for SHA1 implementation with
the SIMD SPE instruction set. With the enhanced instruction
set we can operate on 2 32 bit words in parallel. That helps
reducing the time to calculate W16-W79. For increasing
performance even more the assembler function can compute
hashes for more than one 64 byte input block.

The state of the used SPE registers is preserved via the
stack so we can run from interrupt context. There might
be the case that we interrupt ourselves and push sensitive
data from another context onto our stack. Clear this area
in the stack afterwards to avoid information leakage.

The code is endian independant.
Signed-off-by: NMarkus Stockhausen <stockhausen@collogia.de>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>