If the crypto4xx device is continuously loaded by dm-crypt
and ipsec work, it will start to work intermittent after a
few (between 20-30) seconds, hurting throughput and latency.

This patch contains various stability improvements in order
to fix this issue. So far, the hardware has survived more
than a day without suffering any stalls under the continuous
load.
Signed-off-by: NChristian Lamparter <chunkeey@gmail.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

crypto4xx_core.c:179:6: warning: symbol 'crypto4xx_free_state_record'
	was not declared. Should it be static?
crypto4xx_core.c:331:5: warning: symbol 'crypto4xx_get_n_gd'
	was not declared. Should it be static?
crypto4xx_core.c:652:6: warning: symbol 'crypto4xx_return_pd'
	was not declared. Should it be static?

crypto4xx_return_pd() is not used by anything. Therefore it is removed.
Signed-off-by: NChristian Lamparter <chunkeey@gmail.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

This patch overhauls and fixes code related to crypto4xx_build_pd()

 * crypto4xx_build_pd() did not handle chained source scatterlist.
   This is fixed by replacing the buggy indexed-access of &src[idx]
   with sg_next() in the gather array setup loop.

 * The redundant is_hash, direction, save_iv and pd_ctl members
   in the crypto4xx_ctx struct have been removed.
    - is_hash can be derived from the crypto_async_request parameter.
    - direction is already part of the security association's
      bf.dir bitfield.
    - save_iv is unused.
    - pd_ctl always had the host_ready bit enabled anyway.
      (the hash_final case is rather pointless, since the ahash
       code has been deactivated).

 * make crypto4xx_build_pd()'s caller responsible for converting
   the IV to the LE32 format.

 * change crypto4xx_ahash_update() and crypto4xx_ahash_digest() to
   initialize a temporary destination scatterlist. This allows the
   removal of an ugly cast of req->result (which is a pointer to an
   u8-array) to a scatterlist pointer.

 * change crypto4xx_build_pd() return type to int. After all
   it returns -EINPROGRESS/-EBUSY.

 * fix crypto4xx_build_pd() thread-unsafe sa handling.
Signed-off-by: NChristian Lamparter <chunkeey@gmail.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

The hardware expects that the keys, IVs (and inner/outer hashes)
are in the le32 format.

This patch changes all hardware interface declarations to use
the correct LE32 data format for each field.

In order to pass __CHECK_ENDIAN__ checks, crypto4xx_memcpy_le
has to be honest about the endianness of its parameters.
The function was split and moved to the common crypto4xx_core.h
header. This allows the compiler to generate better code if the
sizes/len is a constant (various *_IV_LEN).

Please note that the hardware isn't consistent with the endiannes
of the save_digest field in the state record struct though.
The hashes produced by GHASH and CBC (for CCM) will be in LE32.
Whereas md5 and sha{1/,256,...} do not need any conversion.
Signed-off-by: NChristian Lamparter <chunkeey@gmail.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Previously, If the crypto4xx driver used all available
security contexts, it would simply refuse new requests
with -EAGAIN. CRYPTO_TFM_REQ_MAY_BACKLOG was ignored.

in case of dm-crypt.c's crypt_convert() function this was
causing the following errors to manifest, if the system was
pushed hard enough:

| EXT4-fs warning (dm-1): ext4_end_bio:314: I/O error -5 writing to ino ..
| EXT4-fs warning (dm-1): ext4_end_bio:314: I/O error -5 writing to ino ..
| EXT4-fs warning (dm-1): ext4_end_bio:314: I/O error -5 writing to ino ..
| JBD2: Detected IO errors while flushing file data on dm-1-8
| Aborting journal on device dm-1-8.
| EXT4-fs error : ext4_journal_check_start:56: Detected aborted journal
| EXT4-fs (dm-1): Remounting filesystem read-only
| EXT4-fs : ext4_writepages: jbd2_start: 2048 pages, inode 498...; err -30

(This did cause corruptions due to failed writes)

To fix this mess, the crypto4xx driver needs to notifiy the
user to slow down. This can be achieved by returning -EBUSY
on requests, once the crypto hardware was falling behind.

Note: -EBUSY has two different meanings. Setting the flag
CRYPTO_TFM_REQ_MAY_BACKLOG implies that the request was
successfully queued, by the crypto driver. To achieve this
requirement, the implementation introduces a threshold check and
adds logic to the completion routines in much the same way as
AMD's Cryptographic Coprocessor (CCP) driver do.

Note2: Tests showed that dm-crypt starved ipsec traffic.
Under load, ipsec links dropped to 0 Kbits/s. This is because
dm-crypt's callback would instantly queue the next request.
In order to not starve ipsec, the driver reserves a small
portion of the available crypto contexts for this purpose.
Signed-off-by: NChristian Lamparter <chunkeey@gmail.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

If crypto4xx is used in conjunction with dm-crypt, the available
ring buffer elements are not enough to handle the load properly.

On an aes-cbc-essiv:sha256 encrypted swap partition the read
performance is abyssal: (tested with hdparm -t)

/dev/mapper/swap_crypt:
 Timing buffered disk reads:  14 MB in  3.68 seconds =   3.81 MB/sec

The patch increases both PPC4XX_NUM_SD and PPC4XX_NUM_PD to 256.
This improves the performance considerably:

/dev/mapper/swap_crypt:
 Timing buffered disk reads: 104 MB in  3.03 seconds =  34.31 MB/sec

Furthermore, PPC4XX_LAST_SD, PPC4XX_LAST_GD and PPC4XX_LAST_PD
can be easily calculated from their respective PPC4XX_NUM_*
constant.
Signed-off-by: NChristian Lamparter <chunkeey@gmail.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

This patch fixes a type mismatch error that I accidentally
introduced when I moved and refactored the dynamic_contents
helpers.
Signed-off-by: NChristian Lamparter <chunkeey@gmail.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

I used aes-cbc as a template for ofb. But sadly I forgot
to update set_key method to crypto4xx_setkey_aes_ofb().

this was caught by the testmgr:
alg: skcipher: Test 1 failed (invalid result) on encr. for ofb-aes-ppc4xx
00000000: 76 49 ab ac 81 19 b2 46 ce e9 8e 9b 12 e9 19 7d
00000010: 50 86 cb 9b 50 72 19 ee 95 db 11 3a 91 76 78 b2
00000020: 73 be d6 b8 e3 c1 74 3b 71 16 e6 9e 22 22 95 16
00000030: 3f f1 ca a1 68 1f ac 09 12 0e ca 30 75 86 e1 a7

With the correct set_key method, the aes-ofb cipher passes the test.

name         : ofb(aes)
driver       : ofb-aes-ppc4xx
module       : crypto4xx
priority     : 300
refcnt       : 1
selftest     : passed
internal     : no
type         : ablkcipher
async        : yes
blocksize    : 16
min keysize  : 16
max keysize  : 32
ivsize       : 16
geniv        : <default>
Signed-off-by: NChristian Lamparter <chunkeey@gmail.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

The hmac_mc parameter of set_dynamic_sa_command_1()
was defined but not used. On closer inspection it
turns out, it was never wired up.
Signed-off-by: NChristian Lamparter <chunkeey@gmail.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

pr_err() messages should terminated with a new-line to avoid
other messages being concatenated onto the end.
Signed-off-by: NArvind Yadav <arvind.yadav.cs@gmail.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

pr_err() messages should terminated with a new-line to avoid
other messages being concatenated onto the end.
Signed-off-by: NArvind Yadav <arvind.yadav.cs@gmail.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

pr_err() messages should terminated with a new-line to avoid
other messages being concatenated onto the end.
Signed-off-by: NArvind Yadav <arvind.yadav.cs@gmail.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

pr_err() messages should terminated with a new-line to avoid
other messages being concatenated onto the end.
Signed-off-by: NArvind Yadav <arvind.yadav.cs@gmail.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

pr_err() messages should terminated with a new-line to avoid
other messages being concatenated onto the end.
Signed-off-by: NArvind Yadav <arvind.yadav.cs@gmail.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

pr_err() messages should terminated with a new-line to avoid
other messages being concatenated onto the end.
Signed-off-by: NArvind Yadav <arvind.yadav.cs@gmail.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

The usage of of_device_get_match_data reduce the code size a bit.
Furthermore, it prevents an improbable dereference when
of_match_device() return NULL.
Signed-off-by: NCorentin Labbe <clabbe.montjoie@gmail.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

The usage of of_device_get_match_data reduce the code size a bit.
Furthermore, it prevents an improbable dereference when
of_match_device() return NULL.
Signed-off-by: NCorentin Labbe <clabbe.montjoie@gmail.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

The usage of of_device_get_match_data reduce the code size a bit.
Furthermore, it prevents an improbable dereference when
of_match_device() return NULL.
Signed-off-by: NCorentin Labbe <clabbe.montjoie@gmail.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Signed-off-by: NAllen Pais <allen.lkml@gmail.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Use the provided mechanism in dmaengine.h to invoke the
completion callback.
Signed-off-by: NGary R Hook <gary.hook@amd.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Clean up the mapped pages and the unmap object once we are done with
it. This enables the final clean-up of the object once the transfer
is complete.
Signed-off-by: NGary R Hook <gary.hook@amd.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Add support to explicity ACK mailbox message
because after sending message we can know
the send status via error attribute of brcm_message.

This is needed to support "txdone_ack" supported in
mailbox controller driver.

Fixes: 9d12ba86 ("crypto: brcm - Add Broadcom SPU driver")
Signed-off-by: NRaveendra Padasalagi <raveendra.padasalagi@broadcom.com>
Reviewed-by: NAnup Patel <anup.patel@broadcom.com>
Reviewed-by: NScott Branden <scott.branden@broadcom.com>
Cc: stable@vger.kernel.org
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Fix checkpatch.pl warnings:

WARNING: void function return statements are not generally useful
FILE: crypto/rmd128.c:218:
FILE: crypto/rmd160.c:261:
FILE: crypto/rmd256.c:233:
FILE: crypto/rmd320.c:280:
FILE: crypto/tcrypt.c:385:
FILE: drivers/crypto/ixp4xx_crypto.c:538:
FILE: drivers/crypto/marvell/cesa.c:81:
FILE: drivers/crypto/ux500/cryp/cryp_core.c:1755:
Signed-off-by: NGeliang Tang <geliangtang@gmail.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Use PTR_ERROR_ZERO rather than if(IS_ERR(...)) + PTR_ERR.
Build successfully.
Signed-off-by: NHimanshu Jha <himanshujha199640@gmail.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Kfree on NULL pointer is a no-op and therefore checking is redundant.
Signed-off-by: NHimanshu Jha <himanshujha199640@gmail.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

kfree on NULL pointer is a no-op and therefore checking it is redundant.
Signed-off-by: NHimanshu Jha <himanshujha199640@gmail.com>
Acked-by: NDavid S. Miller <davem@davemloft.net>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

x86_cpu_id are not supposed to change at runtime. MODULE_DEVICE_TABLE
and x86_match_cpu are working with const x86_cpu_id. So mark the
non-const x86_cpu_id structs as const.
Signed-off-by: NArvind Yadav <arvind.yadav.cs@gmail.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

x86_cpu_id are not supposed to change at runtime. MODULE_DEVICE_TABLE
and x86_match_cpu are working with const x86_cpu_id. So mark the
non-const x86_cpu_id structs as const.
Signed-off-by: NArvind Yadav <arvind.yadav.cs@gmail.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

This patch improves the readability of various functions,
by replacing various void* pointers declarations with
their respective structs *. This makes it possible to go
for the eye-friendly array-indexing methods.
Signed-off-by: NChristian Lamparter <chunkeey@googlemail.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

If one of the later memory allocations in rypto4xx_build_pdr()
fails: dev->pdr (and/or) dev->pdr_uinfo wouldn't be freed.

crypto4xx_build_sdr() has the same issue with dev->sdr.
Signed-off-by: NChristian Lamparter <chunkeey@googlemail.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

scatter_buffer_size is always set to PPC4XX_SD_BUFFER_SIZE.
I don't think there's any point in keeping the variable
around.
Signed-off-by: NChristian Lamparter <chunkeey@googlemail.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

This patch refactors the crypto4xx_copy_pkt_to_dst() to use
scatterwalk_map_and_copy() to copy the processed data between
the crypto engine's scatter ring buffer and the destination
specified by the ablkcipher_request.

This also makes the crypto4xx_fill_one_page() function redundant.
Signed-off-by: NChristian Lamparter <chunkeey@googlemail.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

The crypto engine supports more than just aes-cbc. This patch
enables the remaining AES block cipher modes that pass the
testmanager's test vectors.
Signed-off-by: NChristian Lamparter <chunkeey@googlemail.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

This patch refactors and moves the dynamic_contents helper
functions into the crypto4xx_sa.h header file.

 * get_dynamic_sa_iv_size is no longer needed, as the cryptoapi
   provides the required IV size information as well.

 * refactor the function declarations to use the a pointer to the
   dynamic_sa_contents union, instead of the crypto4xx_ctx.

 * rename get_dynamic_sa_offset_key_field to get_dynamic_sa_key_field.
   It returns the pointer to the key directly.
Signed-off-by: NChristian Lamparter <chunkeey@googlemail.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

The driver had a union dynamic_sa_contents in place that
described the meaning of the bits in the sa_contents
variable.
Signed-off-by: NChristian Lamparter <chunkeey@googlemail.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

crypto4xx_put_pd_to_pdr() already clears the flag.
Signed-off-by: NChristian Lamparter <chunkeey@googlemail.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

All function declarations are "extern" by default, there is no need to
specify it explicitly.

For C99 states in 6.2.2.5:
"If the declaration of an identifier for a function has no
storage-class specifier, its linkage is determined exactly
as if it were declared with the storage-class specifier
extern."
Signed-off-by: NChristian Lamparter <chunkeey@googlemail.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

The security offload function is performed by a cryptographic
engine core attached to the 128-bit PLB (processor local bus)
with builtin DMA and interrupt controllers. This, I think,
satisfies the requirement for the CRYPTO_ALG_KERN_DRIVER_ONLY
flag.
Signed-off-by: NChristian Lamparter <chunkeey@googlemail.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

This patch removes several unused code and definitons
(structs, variables, ...).
Signed-off-by: NChristian Lamparter <chunkeey@googlemail.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

alg entries are only added to the list, after the registration
was successful. If the registration failed, it was never added
to the list in the first place.
Signed-off-by: NChristian Lamparter <chunkeey@googlemail.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>