Add Andy Polyakov's optimized assembly and NEON implementations for
SHA-256/224.

The sha256-armv4.pl script for generating the assembly code is from
OpenSSL commit 51f8d095562f36cdaa6893597b5c609e943b0565.

Compared to sha256-generic these implementations have the following
tcrypt speed improvements on Motorola Nexus 6 (Snapdragon 805):

  bs    b/u      sha256-neon  sha256-asm
  16    16       x1.32        x1.19
  64    16       x1.27        x1.15
  64    64       x1.36        x1.20
  256   16       x1.22        x1.11
  256   64       x1.36        x1.19
  256   256      x1.59        x1.23
  1024  16       x1.21        x1.10
  1024  256      x1.65        x1.23
  1024  1024     x1.76        x1.25
  2048  16       x1.21        x1.10
  2048  256      x1.66        x1.23
  2048  1024     x1.78        x1.25
  2048  2048     x1.79        x1.25
  4096  16       x1.20        x1.09
  4096  256      x1.66        x1.23
  4096  1024     x1.79        x1.26
  4096  4096     x1.82        x1.26
  8192  16       x1.20        x1.09
  8192  256      x1.67        x1.23
  8192  1024     x1.80        x1.26
  8192  4096     x1.85        x1.28
  8192  8192     x1.85        x1.27

Where bs refers to block size and b/u to bytes per update.
Signed-off-by: NSami Tolvanen <samitolvanen@google.com>
Cc: Andy Polyakov <appro@openssl.org>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

This patch makes crypto_unregister_instance take a crypto_instance
instead of a crypto_alg.  This allows us to remove a duplicate
CRYPTO_ALG_INSTANCE check in crypto_unregister_instance.
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

There are multiple problems in crypto_unregister_instance:

1) The cra_refcnt BUG_ON check is racy and can cause crashes.
2) The cra_refcnt check shouldn't exist at all.
3) There is no reference on tmpl to protect the tmpl->free call.

This patch rewrites the function using crypto_remove_spawn which
now morphs into crypto_remove_instance.
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

kmap_atomic() gives only the page address of the input page.
Driver should take care of adding the offset of the scatterlist
within the page to the returned page address.
omap-sham driver is not adding the offset to page and directly operates
on the return vale of kmap_atomic(), because of which the following
error comes when running crypto tests:

00000000: d9 a1 1b 7c aa 90 3b aa 11 ab cb 25 00 b8 ac bf
[    2.338169] 00000010: c1 39 cd ff 48 d0 a8 e2 2b fa 33 a1
[    2.344008] alg: hash: Chunking test 1 failed for omap-sha256

So adding the scatterlist offset to vaddr.
Signed-off-by: NLokesh Vutla <lokeshvutla@ti.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

ERROR:CODE_INDENT: code indent should use tabs where possible
Signed-off-by: NBruce Allan <bruce.w.allan@intel.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

CHECK:COMPARISON_TO_NULL: Comparison to NULL could be written
"!device_reset_wq"
Signed-off-by: NBruce Allan <bruce.w.allan@intel.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

CHECK:BIT_MACRO: Prefer using the BIT macro
Signed-off-by: NBruce Allan <bruce.w.allan@intel.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

CHECK:CONCATENATED_STRING: Concatenated strings should use spaces between
elements
Signed-off-by: NBruce Allan <bruce.w.allan@intel.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Cleanup code to fix the subject checkpatch warnings
Signed-off-by: NBruce Allan <bruce.w.allan@intel.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Signed-off-by: NBruce Allan <bruce.w.allan@intel.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

adt_ctl_drv should be adf_ctl_drv
Signed-off-by: NBruce Allan <bruce.w.allan@intel.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

For cases where total length of an input SGs is not same as
length of the input data for encryption, omap-aes driver
crashes. This happens in the case when IPsec is trying to use
omap-aes driver.

To avoid this, we copy all the pages from the input SG list
into a contiguous buffer and prepare a single element SG list
for this buffer with length as the total bytes to crypt, which is
similar thing that is done in case of unaligned lengths.

Fixes: 6242332f ("crypto: omap-aes - Add support for cases of unaligned lengths")
Signed-off-by: NLokesh Vutla <lokeshvutla@ti.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

omap_sham_handle_queue() can be called as part of done_task tasklet.
During this its atomic and any calls to pm functions cannot sleep.

But there is a call to pm_runtime_get_sync() (which can sleep) in
omap_sham_handle_queue(), because of which the following appears:
" [  116.169969] BUG: scheduling while atomic: kworker/0:2/2676/0x00000100"

Add pm_runtime_irq_safe() to avoid this.
Signed-off-by: NLokesh Vutla <lokeshvutla@ti.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Flag all Multi buffer SHA1 helper ciphers as internal ciphers
to prevent them from being called by normal users.
Signed-off-by: NStephan Mueller <smueller@chronox.de>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

The mcryptd is used as a wrapper around internal ciphers. Therefore,
the mcryptd must process the internal cipher by marking mcryptd as
internal if the underlying cipher is an internal cipher.
Signed-off-by: NStephan Mueller <smueller@chronox.de>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Flag all 64 bit ARMv8 AES helper ciphers as internal ciphers to
prevent them from being called by normal users.
Signed-off-by: NStephan Mueller <smueller@chronox.de>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Flag all ARMv8 AES helper ciphers as internal ciphers to prevent
them from being called by normal users.
Signed-off-by: NStephan Mueller <smueller@chronox.de>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Flag all NEON bit sliced AES helper ciphers as internal ciphers to
prevent them from being called by normal users.
Signed-off-by: NStephan Mueller <smueller@chronox.de>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Flag all Twofish AVX helper ciphers as internal ciphers to prevent
them from being called by normal users.
Signed-off-by: NStephan Mueller <smueller@chronox.de>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Flag all Serpent SSE2 helper ciphers as internal ciphers to prevent
them from being called by normal users.
Signed-off-by: NStephan Mueller <smueller@chronox.de>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Flag all Serpent AVX helper ciphers as internal ciphers to prevent
them from being called by normal users.
Signed-off-by: NStephan Mueller <smueller@chronox.de>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Flag all Serpent AVX2 helper ciphers as internal ciphers to prevent
them from being called by normal users.
Signed-off-by: NStephan Mueller <smueller@chronox.de>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Flag all CAST6 helper ciphers as internal ciphers to prevent them
from being called by normal users.
Signed-off-by: NStephan Mueller <smueller@chronox.de>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Flag all AVX Camellia helper ciphers as internal ciphers to prevent
them from being called by normal users.
Signed-off-by: NStephan Mueller <smueller@chronox.de>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Flag all CAST5 helper ciphers as internal ciphers to prevent them
from being called by normal users.
Signed-off-by: NStephan Mueller <smueller@chronox.de>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Flag all AES-NI Camellia helper ciphers as internal ciphers to
prevent them from being called by normal users.
Signed-off-by: NStephan Mueller <smueller@chronox.de>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Flag all GHASH ARMv8 vmull.p64 helper ciphers as internal ciphers
to prevent them from being called by normal users.
Signed-off-by: NStephan Mueller <smueller@chronox.de>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Flag all ash clmulni helper ciphers as internal ciphers to prevent them
from being called by normal users.
Signed-off-by: NStephan Mueller <smueller@chronox.de>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Flag all AES-NI helper ciphers as internal ciphers to prevent them from
being called by normal users.
Signed-off-by: NStephan Mueller <smueller@chronox.de>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

With ciphers that now cannot be accessed via the kernel crypto API,
callers shall be able to identify the ciphers that are not callable. The
/proc/crypto file is added a boolean field identifying that such
internal ciphers.
Signed-off-by: NStephan Mueller <smueller@chronox.de>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

The cryptd is used as a wrapper around internal ciphers. Therefore, the
cryptd must process the internal cipher by marking cryptd as internal if
the underlying cipher is an internal cipher.
Signed-off-by: NStephan Mueller <smueller@chronox.de>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Allocate the ciphers irrespectively if they are marked as internal
or not. As all ciphers, including the internal ciphers will be
processed by the testmgr, it needs to be able to allocate those
ciphers.
Signed-off-by: NStephan Mueller <smueller@chronox.de>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Several hardware related cipher implementations are implemented as
follows: a "helper" cipher implementation is registered with the
kernel crypto API.

Such helper ciphers are never intended to be called by normal users. In
some cases, calling them via the normal crypto API may even cause
failures including kernel crashes. In a normal case, the "wrapping"
ciphers that use the helpers ensure that these helpers are invoked
such that they cannot cause any calamity.

Considering the AF_ALG user space interface, unprivileged users can
call all ciphers registered with the crypto API, including these
helper ciphers that are not intended to be called directly. That
means, with AF_ALG user space may invoke these helper ciphers
and may cause undefined states or side effects.

To avoid any potential side effects with such helpers, the patch
prevents the helpers to be called directly. A new cipher type
flag is added: CRYPTO_ALG_INTERNAL. This flag shall be used
to mark helper ciphers. These ciphers can only be used if the
caller invoke the cipher with CRYPTO_ALG_INTERNAL in the type and
mask field.
Signed-off-by: NStephan Mueller <smueller@chronox.de>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

This fixes a bug in the new v8 Crypto Extensions GHASH code
that only manifests itself in big-endian mode.
Signed-off-by: NArd Biesheuvel <ard.biesheuvel@linaro.org>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

"hdev->req->nbytes" is an unsigned int so we so we lose the upper 3 bits
to the shift wrap bug.
Signed-off-by: NDan Carpenter <dan.carpenter@oracle.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

GCC complains about that %u is the wrong format string for size_t and
also that "ret" is unused.
Signed-off-by: NDan Carpenter <dan.carpenter@oracle.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Convert pr_info() and pr_err() log messages to dev_info() and dev_err(),
respectively, where able.  This adds the module name and PCI B:D:F to
indicate which QAT device generated the log message.  The "QAT:" is removed
from these log messages as that is now unnecessary.  A few of these log
messages have additional spelling/contextual fixes.
Signed-off-by: NBruce Allan <bruce.w.allan@intel.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Signed-off-by: NBruce Allan <bruce.w.allan@intel.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

This define is a duplicate of the one in ./include/linux/pci_ids.h
Signed-off-by: NBruce Allan <bruce.w.allan@intel.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

The Kconfig entry for CRYPTO_DEV_IMGTEC_HASH incorrectly selects
CRYPTO_SHA224, which does not exist (and is covered by CRYPTO_SHA256
which covers both 224 and 256). Remove it.

Also correct typo CRYPTO_ALG_API to be CRYPTO_ALGPI.
Reported-by: NValentin Rothberg <valentinrothberg@gmail.com>
Signed-off-by: NJames Hartley <james.hartley@imgtec.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>