Thanks to the big overhaul of crypto4xx_build_pd(), the request-local
sa_in, sa_out and state_record allocation can be simplified.

There's no need to setup any dma coherent memory anymore and
much of the support code can be removed.
Signed-off-by: NChristian Lamparter <chunkeey@gmail.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

If the crypto4xx device is continuously loaded by dm-crypt
and ipsec work, it will start to work intermittent after a
few (between 20-30) seconds, hurting throughput and latency.

This patch contains various stability improvements in order
to fix this issue. So far, the hardware has survived more
than a day without suffering any stalls under the continuous
load.
Signed-off-by: NChristian Lamparter <chunkeey@gmail.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

crypto4xx_core.c:179:6: warning: symbol 'crypto4xx_free_state_record'
	was not declared. Should it be static?
crypto4xx_core.c:331:5: warning: symbol 'crypto4xx_get_n_gd'
	was not declared. Should it be static?
crypto4xx_core.c:652:6: warning: symbol 'crypto4xx_return_pd'
	was not declared. Should it be static?

crypto4xx_return_pd() is not used by anything. Therefore it is removed.
Signed-off-by: NChristian Lamparter <chunkeey@gmail.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

This patch overhauls and fixes code related to crypto4xx_build_pd()

 * crypto4xx_build_pd() did not handle chained source scatterlist.
   This is fixed by replacing the buggy indexed-access of &src[idx]
   with sg_next() in the gather array setup loop.

 * The redundant is_hash, direction, save_iv and pd_ctl members
   in the crypto4xx_ctx struct have been removed.
    - is_hash can be derived from the crypto_async_request parameter.
    - direction is already part of the security association's
      bf.dir bitfield.
    - save_iv is unused.
    - pd_ctl always had the host_ready bit enabled anyway.
      (the hash_final case is rather pointless, since the ahash
       code has been deactivated).

 * make crypto4xx_build_pd()'s caller responsible for converting
   the IV to the LE32 format.

 * change crypto4xx_ahash_update() and crypto4xx_ahash_digest() to
   initialize a temporary destination scatterlist. This allows the
   removal of an ugly cast of req->result (which is a pointer to an
   u8-array) to a scatterlist pointer.

 * change crypto4xx_build_pd() return type to int. After all
   it returns -EINPROGRESS/-EBUSY.

 * fix crypto4xx_build_pd() thread-unsafe sa handling.
Signed-off-by: NChristian Lamparter <chunkeey@gmail.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

The hardware expects that the keys, IVs (and inner/outer hashes)
are in the le32 format.

This patch changes all hardware interface declarations to use
the correct LE32 data format for each field.

In order to pass __CHECK_ENDIAN__ checks, crypto4xx_memcpy_le
has to be honest about the endianness of its parameters.
The function was split and moved to the common crypto4xx_core.h
header. This allows the compiler to generate better code if the
sizes/len is a constant (various *_IV_LEN).

Please note that the hardware isn't consistent with the endiannes
of the save_digest field in the state record struct though.
The hashes produced by GHASH and CBC (for CCM) will be in LE32.
Whereas md5 and sha{1/,256,...} do not need any conversion.
Signed-off-by: NChristian Lamparter <chunkeey@gmail.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Previously, If the crypto4xx driver used all available
security contexts, it would simply refuse new requests
with -EAGAIN. CRYPTO_TFM_REQ_MAY_BACKLOG was ignored.

in case of dm-crypt.c's crypt_convert() function this was
causing the following errors to manifest, if the system was
pushed hard enough:

| EXT4-fs warning (dm-1): ext4_end_bio:314: I/O error -5 writing to ino ..
| EXT4-fs warning (dm-1): ext4_end_bio:314: I/O error -5 writing to ino ..
| EXT4-fs warning (dm-1): ext4_end_bio:314: I/O error -5 writing to ino ..
| JBD2: Detected IO errors while flushing file data on dm-1-8
| Aborting journal on device dm-1-8.
| EXT4-fs error : ext4_journal_check_start:56: Detected aborted journal
| EXT4-fs (dm-1): Remounting filesystem read-only
| EXT4-fs : ext4_writepages: jbd2_start: 2048 pages, inode 498...; err -30

(This did cause corruptions due to failed writes)

To fix this mess, the crypto4xx driver needs to notifiy the
user to slow down. This can be achieved by returning -EBUSY
on requests, once the crypto hardware was falling behind.

Note: -EBUSY has two different meanings. Setting the flag
CRYPTO_TFM_REQ_MAY_BACKLOG implies that the request was
successfully queued, by the crypto driver. To achieve this
requirement, the implementation introduces a threshold check and
adds logic to the completion routines in much the same way as
AMD's Cryptographic Coprocessor (CCP) driver do.

Note2: Tests showed that dm-crypt starved ipsec traffic.
Under load, ipsec links dropped to 0 Kbits/s. This is because
dm-crypt's callback would instantly queue the next request.
In order to not starve ipsec, the driver reserves a small
portion of the available crypto contexts for this purpose.
Signed-off-by: NChristian Lamparter <chunkeey@gmail.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

If crypto4xx is used in conjunction with dm-crypt, the available
ring buffer elements are not enough to handle the load properly.

On an aes-cbc-essiv:sha256 encrypted swap partition the read
performance is abyssal: (tested with hdparm -t)

/dev/mapper/swap_crypt:
 Timing buffered disk reads:  14 MB in  3.68 seconds =   3.81 MB/sec

The patch increases both PPC4XX_NUM_SD and PPC4XX_NUM_PD to 256.
This improves the performance considerably:

/dev/mapper/swap_crypt:
 Timing buffered disk reads: 104 MB in  3.03 seconds =  34.31 MB/sec

Furthermore, PPC4XX_LAST_SD, PPC4XX_LAST_GD and PPC4XX_LAST_PD
can be easily calculated from their respective PPC4XX_NUM_*
constant.
Signed-off-by: NChristian Lamparter <chunkeey@gmail.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

This patch fixes a type mismatch error that I accidentally
introduced when I moved and refactored the dynamic_contents
helpers.
Signed-off-by: NChristian Lamparter <chunkeey@gmail.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

I used aes-cbc as a template for ofb. But sadly I forgot
to update set_key method to crypto4xx_setkey_aes_ofb().

this was caught by the testmgr:
alg: skcipher: Test 1 failed (invalid result) on encr. for ofb-aes-ppc4xx
00000000: 76 49 ab ac 81 19 b2 46 ce e9 8e 9b 12 e9 19 7d
00000010: 50 86 cb 9b 50 72 19 ee 95 db 11 3a 91 76 78 b2
00000020: 73 be d6 b8 e3 c1 74 3b 71 16 e6 9e 22 22 95 16
00000030: 3f f1 ca a1 68 1f ac 09 12 0e ca 30 75 86 e1 a7

With the correct set_key method, the aes-ofb cipher passes the test.

name         : ofb(aes)
driver       : ofb-aes-ppc4xx
module       : crypto4xx
priority     : 300
refcnt       : 1
selftest     : passed
internal     : no
type         : ablkcipher
async        : yes
blocksize    : 16
min keysize  : 16
max keysize  : 32
ivsize       : 16
geniv        : <default>
Signed-off-by: NChristian Lamparter <chunkeey@gmail.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

The hmac_mc parameter of set_dynamic_sa_command_1()
was defined but not used. On closer inspection it
turns out, it was never wired up.
Signed-off-by: NChristian Lamparter <chunkeey@gmail.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

The code is simplified by using two __be64 values for the operation
instead of using two arrays of u8. This allows to get rid of the memory
alignment code. In addition, the crypto_xor can be replaced with a
native XOR operation. Finally, the definition of the variables is
re-arranged such that the data structures come before simple variables
to potentially reduce memory space.
Signed-off-by: NStephan Mueller <smueller@chronox.de>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

DH_KPP_SECRET_MIN_SIZE and dh_data_size() are both returning
unsigned values.
Signed-off-by: NTudor Ambarus <tudor.ambarus@microchip.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

p->key_size, p->p_size, p->g_size are all of unsigned int type.
Signed-off-by: NTudor Ambarus <tudor.ambarus@microchip.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

ECDH_KPP_SECRET_MIN_SIZE and params->key_size are both returning
unsigned values.
Signed-off-by: NTudor Ambarus <tudor.ambarus@microchip.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

pr_err() messages should terminated with a new-line to avoid
other messages being concatenated onto the end.
Signed-off-by: NArvind Yadav <arvind.yadav.cs@gmail.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

pr_err() messages should terminated with a new-line to avoid
other messages being concatenated onto the end.
Signed-off-by: NArvind Yadav <arvind.yadav.cs@gmail.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

pr_err() messages should terminated with a new-line to avoid
other messages being concatenated onto the end.
Signed-off-by: NArvind Yadav <arvind.yadav.cs@gmail.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

pr_err() messages should terminated with a new-line to avoid
other messages being concatenated onto the end.
Signed-off-by: NArvind Yadav <arvind.yadav.cs@gmail.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

pr_err() messages should terminated with a new-line to avoid
other messages being concatenated onto the end.
Signed-off-by: NArvind Yadav <arvind.yadav.cs@gmail.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

pr_err() messages should terminated with a new-line to avoid
other messages being concatenated onto the end.
Signed-off-by: NArvind Yadav <arvind.yadav.cs@gmail.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

If the cipher name does not start with 'ecb(' we should bail out, as done
in the 'create()' function in 'crypto/xts.c'.

Fixes: 700cb3f5 ("crypto: lrw - Convert to skcipher")
Signed-off-by: NChristophe JAILLET <christophe.jaillet@wanadoo.fr>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

All error handling paths 'goto err_drop_spawn' except this one.
In order to avoid some resources leak, we should do it as well here.

Fixes: 700cb3f5 ("crypto: lrw - Convert to skcipher")
Signed-off-by: NChristophe JAILLET <christophe.jaillet@wanadoo.fr>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

The usage of of_device_get_match_data reduce the code size a bit.
Furthermore, it prevents an improbable dereference when
of_match_device() return NULL.
Signed-off-by: NCorentin Labbe <clabbe.montjoie@gmail.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

The usage of of_device_get_match_data reduce the code size a bit.
Furthermore, it prevents an improbable dereference when
of_match_device() return NULL.
Signed-off-by: NCorentin Labbe <clabbe.montjoie@gmail.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

The usage of of_device_get_match_data reduce the code size a bit.
Furthermore, it prevents an improbable dereference when
of_match_device() return NULL.
Signed-off-by: NCorentin Labbe <clabbe.montjoie@gmail.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

The variable len is set to zero, never read and then later updated
to p - name, so clearly the zero'ing of len is redundant and
can be removed.

Detected by clang scan-build:
" warning: Value stored to 'len' is never read"
Signed-off-by: NColin Ian King <colin.king@canonical.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Signed-off-by: NAllen Pais <allen.lkml@gmail.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

If the algorithm we're parallelizing is asynchronous we might change
CPUs between padata_do_parallel() and padata_do_serial(). However, we
don't expect this to happen as we need to enqueue the padata object into
the per-cpu reorder queue we took it from, i.e. the same-cpu's parallel
queue.

Ensure we're not switching CPUs for a given padata object by tracking
the CPU within the padata object. If the serial callback gets called on
the wrong CPU, defer invoking padata_reorder() via a kernel worker on
the CPU we're expected to run on.
Signed-off-by: NMathias Krause <minipli@googlemail.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

The reorder timer function runs on the CPU where the timer interrupt was
handled which is not necessarily one of the CPUs of the 'pcpu' CPU mask
set.

Ensure the padata_reorder() callback runs on the correct CPU, which is
one in the 'pcpu' CPU mask set and, preferrably, the next expected one.
Do so by comparing the current CPU with the expected target CPU. If they
match, call padata_reorder() right away. If they differ, schedule a work
item on the target CPU that does the padata_reorder() call for us.
Signed-off-by: NMathias Krause <minipli@googlemail.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

The parallel queue per-cpu data structure gets initialized only for CPUs
in the 'pcpu' CPU mask set. This is not sufficient as the reorder timer
may run on a different CPU and might wrongly decide it's the target CPU
for the next reorder item as per-cpu memory gets memset(0) and we might
be waiting for the first CPU in cpumask.pcpu, i.e. cpu_index 0.

Make the '__this_cpu_read(pd->pqueue->cpu_index) == next_queue->cpu_index'
compare in padata_get_next() fail in this case by initializing the
cpu_index member of all per-cpu parallel queues. Use -1 for unused ones.
Signed-off-by: NMathias Krause <minipli@googlemail.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

In 32-bit mode, the x86 architecture can hold full 32-bit pointers.
Therefore, the code that copies the current address to the %ecx register
and uses %ecx-relative addressing is useless, we could just use absolute
addressing.

The processors have a stack of return addresses for branch prediction. If
we use a call instruction and pop the return address, it desynchronizes
the return stack and causes branch prediction misses.

This patch also moves the data to the .rodata section.
Signed-off-by: NMikulas Patocka <mpatocka@redhat.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Use the provided mechanism in dmaengine.h to invoke the
completion callback.
Signed-off-by: NGary R Hook <gary.hook@amd.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Clean up the mapped pages and the unmap object once we are done with
it. This enables the final clean-up of the object once the transfer
is complete.
Signed-off-by: NGary R Hook <gary.hook@amd.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Add support to explicity ACK mailbox message
because after sending message we can know
the send status via error attribute of brcm_message.

This is needed to support "txdone_ack" supported in
mailbox controller driver.

Fixes: 9d12ba86 ("crypto: brcm - Add Broadcom SPU driver")
Signed-off-by: NRaveendra Padasalagi <raveendra.padasalagi@broadcom.com>
Reviewed-by: NAnup Patel <anup.patel@broadcom.com>
Reviewed-by: NScott Branden <scott.branden@broadcom.com>
Cc: stable@vger.kernel.org
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Fix checkpatch.pl warnings:

WARNING: void function return statements are not generally useful
FILE: crypto/rmd128.c:218:
FILE: crypto/rmd160.c:261:
FILE: crypto/rmd256.c:233:
FILE: crypto/rmd320.c:280:
FILE: crypto/tcrypt.c:385:
FILE: drivers/crypto/ixp4xx_crypto.c:538:
FILE: drivers/crypto/marvell/cesa.c:81:
FILE: drivers/crypto/ux500/cryp/cryp_core.c:1755:
Signed-off-by: NGeliang Tang <geliangtang@gmail.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Use PTR_ERROR_ZERO rather than if(IS_ERR(...)) + PTR_ERR.
Build successfully.
Signed-off-by: NHimanshu Jha <himanshujha199640@gmail.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Kfree on NULL pointer is a no-op and therefore checking is redundant.
Signed-off-by: NHimanshu Jha <himanshujha199640@gmail.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

kfree on NULL pointer is a no-op and therefore checking it is redundant.
Signed-off-by: NHimanshu Jha <himanshujha199640@gmail.com>
Acked-by: NDavid S. Miller <davem@davemloft.net>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

x86_cpu_id are not supposed to change at runtime. MODULE_DEVICE_TABLE
and x86_match_cpu are working with const x86_cpu_id. So mark the
non-const x86_cpu_id structs as const.
Signed-off-by: NArvind Yadav <arvind.yadav.cs@gmail.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

x86_cpu_id are not supposed to change at runtime. MODULE_DEVICE_TABLE
and x86_match_cpu are working with const x86_cpu_id. So mark the
non-const x86_cpu_id structs as const.
Signed-off-by: NArvind Yadav <arvind.yadav.cs@gmail.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>