The existing test suite to calculate the speed of the SHA algorithms
assumes serial (single buffer)) computation of data. With the SHA
multibuffer algorithms, we work on 8 lanes of data in parallel. Hence,
the need to introduce a new test suite to calculate the speed for these
algorithms.
Signed-off-by: NMegha Dey <megha.dey@linux.intel.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

This patch introduces the assembly routines to do SHA256 computation
on buffers belonging to several jobs at once.  The assembly routines
are optimized with AVX2 instructions that have 8 data lanes and using
AVX2 registers.
Signed-off-by: NMegha Dey <megha.dey@linux.intel.com>
Reviewed-by: NFenghua Yu <fenghua.yu@intel.com>
Reviewed-by: NTim Chen <tim.c.chen@linux.intel.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

This patch introduces the data structures and prototypes of
functions needed for computing SHA256 hash using multi-buffer.
Included are the structures of the multi-buffer SHA256 job,
job scheduler in C and x86 assembly.
Signed-off-by: NMegha Dey <megha.dey@linux.intel.com>
Reviewed-by: NFenghua Yu <fenghua.yu@intel.com>
Reviewed-by: NTim Chen <tim.c.chen@linux.intel.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

This patch introduces the routines used to submit and flush buffers
belonging to SHA256 crypto jobs to the SHA256 multibuffer algorithm. It
is implemented mostly in assembly optimized with AVX2 instructions.
Signed-off-by: NMegha Dey <megha.dey@linux.intel.com>
Reviewed-by: NFenghua Yu <fenghua.yu@intel.com>
Reviewed-by: NTim Chen <tim.c.chen@linux.intel.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Add the config CRYPTO_SHA256_MB which will enable the computation using the
SHA256 multi-buffer algorithm.
Signed-off-by: NMegha Dey <megha.dey@linux.intel.com>
Reviewed-by: NFenghua Yu <fenghua.yu@intel.com>
Reviewed-by: NTim Chen <tim.c.chen@linux.intel.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

This patch introduces the multi-buffer job manager which is responsible for
submitting scatter-gather buffers from several SHA256 jobs to the
multi-buffer algorithm. It also contains the flush routine to that's
called by the crypto daemon to complete the job when no new jobs arrive
before the deadline of maximum latency of a SHA256 crypto job.

The SHA256 multi-buffer crypto algorithm is defined and initialized in
this patch.
Signed-off-by: NMegha Dey <megha.dey@linux.intel.com>
Reviewed-by: NFenghua Yu <fenghua.yu@intel.com>
Reviewed-by: NTim Chen <tim.c.chen@linux.intel.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Document the binding used by the Broadcom BCM5301x (Northstar) SoC
random number generator.
Signed-off-by: NFlorian Fainelli <f.fainelli@gmail.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

There is another ecdh_shared_secret in net/bluetooth/ecc.c

Fixes: 3c4b2390 ("crypto: ecdh - Add ECDH software support")
Signed-off-by: NStephen Rothwell <sfr@canb.auug.org.au>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

The Broadcom BCM5301x SoCs (Northstar) utilize the same random number
generator peripheral as Northstar Plus and BCM2835, but just like the
NSP SoC, we need to enable the interrupt.
Signed-off-by: NFlorian Fainelli <f.fainelli@gmail.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

As part of the Y2038 development, __getnstimeofday is not supposed to be
used any more. It is now replaced with ktime_get_ns. The Jitter RNG uses
the time stamp to measure the execution time of a given code path and
tries to detect variations in the execution time. Therefore, the only
requirement the Jitter RNG has, is a sufficient high resolution to
detect these variations.

The change was tested on x86 to show an identical behavior as RDTSC. The
used test code simply measures the execution time of the heart of the
RNG:

        jent_get_nstime(&time);
        jent_memaccess(ec, min);
        jent_fold_time(NULL, time, &folded, min);
        jent_get_nstime(&time2);
        return ((time2 - time));
Signed-off-by: NStephan Mueller <smueller@chronox.de>
Acked-by: NArnd Bergmann <arnd@arndb.de>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

This patch replaces use of the obsolete blkcipher with skcipher.
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>
Acked-by: NDavid Howells <dhowells@redhat.com>

Adds software fallback support for small crypto requests. In these cases,
it is undesirable to use DMA, as setting it up itself is rather heavy
operation. Gives about 40% extra performance in ipsec usecase.
Signed-off-by: NBin Liu <b-liu@ti.com>
[t-kristo@ti.com: dropped the extra traces, updated some comments
 on the code]
Signed-off-by: NTero Kristo <t-kristo@ti.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

The extra call to dmaengine_terminate_all is not needed, as the DMA
is not running at this point. This improves performance slightly.
Signed-off-by: NLokesh Vutla <lokeshvutla@ti.com>
Signed-off-by: NTero Kristo <t-kristo@ti.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Change crypto queue size from 1 to 10 for omap SHA driver. This should
allow clients to enqueue requests more effectively to avoid serializing
whole crypto sequences, giving extra performance.
Signed-off-by: NTero Kristo <t-kristo@ti.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Calling runtime PM API for every block causes serious performance hit to
crypto operations that are done on a long buffer. As crypto is performed
on a page boundary, encrypting large buffers can cause a series of crypto
operations divided by page. The runtime PM API is also called those many
times.

Convert the driver to use runtime_pm autosuspend instead, with a default
timeout value of 1 second. This results in upto ~50% speedup.
Signed-off-by: NTero Kristo <t-kristo@ti.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

* Implement ECDH under kpp API
 * Provide ECC software support for curve P-192 and
   P-256.
 * Add kpp test for ECDH with data generated by OpenSSL
Signed-off-by: NSalvatore Benedetto <salvatore.benedetto@intel.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

* Implement MPI based Diffie-Hellman under kpp API
 * Test provided uses data generad by OpenSSL
Signed-off-by: NSalvatore Benedetto <salvatore.benedetto@intel.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Add key-agreement protocol primitives (kpp) API which allows to
implement primitives required by protocols such as DH and ECDH.
The API is composed mainly by the following functions
 * set_secret() - It allows the user to set his secret, also
   referred to as his private key, along with the parameters
   known to both parties involved in the key-agreement session.
 * generate_public_key() - It generates the public key to be sent to
   the other counterpart involved in the key-agreement session. The
   function has to be called after set_params() and set_secret()
 * generate_secret() - It generates the shared secret for the session

Other functions such as init() and exit() are provided for allowing
cryptographic hardware to be inizialized properly before use
Signed-off-by: NSalvatore Benedetto <salvatore.benedetto@intel.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Herbert wants the sha1-mb algorithm to have an async implementation:
https://lkml.org/lkml/2016/4/5/286.
Currently, sha1-mb uses an async interface for the outer algorithm
and a sync interface for the inner algorithm. This patch introduces
a async interface for even the inner algorithm.
Signed-off-by: NMegha Dey <megha.dey@linux.intel.com>
Signed-off-by: NTim Chen <tim.c.chen@linux.intel.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

This patch fixes an old bug where requests can be reordered because
some are processed by cryptd while others are processed directly
in softirq context.

The fix is to always postpone to cryptd if there are currently
requests outstanding from the same tfm.

This patch also removes the redundant use of cryptd in the async
init function as init never touches the FPU.
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

This patch fixes an old bug where requests can be reordered because
some are processed by cryptd while others are processed directly
in softirq context.

The fix is to always postpone to cryptd if there are currently
requests outstanding from the same tfm.

This patch also removes the redundant use of cryptd in the async
init function as init never touches the FPU.
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

This patch fixes an old bug where requests can be reordered because
some are processed by cryptd while others are processed directly
in softirq context.

The fix is to always postpone to cryptd if there are currently
requests outstanding from the same tfm.
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

This patch fixes an old bug where gcm requests can be reordered
because some are processed by cryptd while others are processed
directly in softirq context.

The fix is to always postpone to cryptd if there are currently
requests outstanding from the same tfm.
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

This patch adds helpers to check whether a given tfm is currently
queued.  This is meant to be used by ablk_helper and similar
entities to ensure that no reordering is introduced because of
requests queued in cryptd with respect to requests being processed
in softirq context.

The per-cpu queue length limit is also increased to 1000 in line
with network limits.
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Now that crypto requests are chained together at the DMA level, we
increase the size of the crypto queue for each engine. The result is
that as the backlog list is reached later, it does not stop the crypto
stack from sending asychronous requests, so more cryptographic tasks
are processed by the engines.
Signed-off-by: NRomain Perier <romain.perier@free-electrons.com>
Acked-by: NBoris Brezillon <boris.brezillon@free-electrons.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

The Cryptographic Engines and Security Accelerators (CESA) supports the
Multi-Packet Chain Mode. With this mode enabled, multiple tdma requests
can be chained and processed by the hardware without software
intervention. This mode was already activated, however the crypto
requests were not chained together. By doing so, we reduce significantly
the number of IRQs. Instead of being interrupted at the end of each
crypto request, we are interrupted at the end of the last cryptographic
request processed by the engine.

This commits re-factorizes the code, changes the code architecture and
adds the required data structures to chain cryptographic requests
together before sending them to an engine (stopped or possibly already
running).
Signed-off-by: NRomain Perier <romain.perier@free-electrons.com>
Acked-by: NBoris Brezillon <boris.brezillon@free-electrons.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

This commits adds support for fine grained load balancing on
multi-engine IPs. The engine is pre-selected based on its current load
and on the weight of the crypto request that is about to be processed.
The global crypto queue is also moved to each engine. These changes are
required to allow chaining crypto requests at the DMA level. By using
a crypto queue per engine, we make sure that we keep the state of the
tdma chain synchronized with the crypto queue. We also reduce contention
on 'cesa_dev->lock' and improve parallelism.
Signed-off-by: NRomain Perier <romain.perier@free-electrons.com>
Acked-by: NBoris Brezillon <boris.brezillon@free-electrons.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Currently the crypto requests were sent to engines sequentially.
This commit moves the SRAM I/O operations from the prepare to the step
functions. It provides flexibility for future works and allow to prepare
a request while the engine is running.
Signed-off-by: NRomain Perier <romain.perier@free-electrons.com>
Acked-by: NBoris Brezillon <boris.brezillon@free-electrons.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

So far, the 'process' operation was used to check if the current request
was correctly handled by the engine, if it was the case it copied
information from the SRAM to the main memory. Now, we split this
operation. We keep the 'process' operation, which still checks if the
request was correctly handled by the engine or not, then we add a new
operation for completion. The 'complete' method copies the content of
the SRAM to memory. This will soon become useful if we want to call
the process and the complete operations from different locations
depending on the type of the request (different cleanup logic).
Signed-off-by: NRomain Perier <romain.perier@free-electrons.com>
Acked-by: NBoris Brezillon <boris.brezillon@free-electrons.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Currently, the only way to access the tdma chain is to use the 'req'
union from a mv_cesa_{ablkcipher,ahash}. This will soon become a problem
if we want to handle the TDMA chaining vs standard/non-DMA processing in
a generic way (with generic functions at the cesa.c level detecting
whether the request should be queued at the DMA level or not). Hence the
decision to move the chain field a the mv_cesa_req level at the expense
of adding 2 void * fields to all request contexts (including non-DMA
ones) and to remove the type completly. To limit the overhead, we get
rid of the type field, which can now be deduced from the req->chain.first
value. Once these changes are done the union is no longer needed, so
remove it and move mv_cesa_ablkcipher_std_req and mv_cesa_req
to mv_cesa_ablkcipher_req directly. There are also no needs to keep the
'base' field into the union of mv_cesa_ahash_req, so move it into the
upper structure.
Signed-off-by: NRomain Perier <romain.perier@free-electrons.com>
Acked-by: NBoris Brezillon <boris.brezillon@free-electrons.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Add a TDMA descriptor at the end of the request for copying the
output IV vector via a DMA transfer. This is a good way for offloading
as much as processing as possible to the DMA and the crypto engine.
This is also required for processing multiple cipher requests
in chained mode, otherwise the content of the IV vector would be
overwritten by the last processed request.
Signed-off-by: NRomain Perier <romain.perier@free-electrons.com>
Acked-by: NBoris Brezillon <boris.brezillon@free-electrons.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

So far, the way that the type of a TDMA operation was checked was wrong.
We have to use the type mask in order to get the right part of the flag
containing the type of the operation.
Signed-off-by: NRomain Perier <romain.perier@free-electrons.com>
Acked-by: NBoris Brezillon <boris.brezillon@free-electrons.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Add a BUG_ON() call when the driver tries to launch a crypto request
while the engine is still processing the previous one. This replaces
a silent system hang by a verbose kernel panic with the associated
backtrace to let the user know that something went wrong in the CESA
driver.
Signed-off-by: NRomain Perier <romain.perier@free-electrons.com>
Acked-by: NBoris Brezillon <boris.brezillon@free-electrons.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Adding a macro constant to be used for the size of the crypto queue,
instead of using a numeric value directly. It will be easier to
maintain in case we add more than one crypto queue of the same size.
Signed-off-by: NRomain Perier <romain.perier@free-electrons.com>
Acked-by: NBoris Brezillon <boris.brezillon@free-electrons.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

On 16-byte requests the optimised version is actually slower than
the generic code, so we should simply use that instead.
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Cheers,

This patch commit eed1e1af as
it is only a workaround for the real bug and the proper fix has
now been applied as 055ddaac
("crypto: user - re-add size check for CRYPTO_MSG_GETALG").
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

We accidentally return PTR_ERR(NULL) which is success but we should
return -ENOMEM.

Fixes: 35591285 ('crypto: drbg - use CTR AES instead of ECB AES')
Signed-off-by: NDan Carpenter <dan.carpenter@oracle.com>
Acked-by: NStephan Mueller <smueller@chronox.de>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Added support for SHA-3 algorithm test's
in tcrypt module and related test vectors.
Signed-off-by: NRaveendra Padasalagi <raveendra.padasalagi@broadcom.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

This patch adds the implementation of SHA3 algorithm
in software and it's based on original implementation
pushed in patch https://lwn.net/Articles/518415/ with
additional changes to match the padding rules specified
in SHA-3 specification.
Signed-off-by: NJeff Garzik <jgarzik@redhat.com>
Signed-off-by: NRaveendra Padasalagi <raveendra.padasalagi@broadcom.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

EXTRA_CFLAGS is still supported but its usage is deprecated.
Signed-off-by: NTudor Ambarus <tudor-dan.ambarus@nxp.com>
Reviewed-by: NHoria Geantă <horia.geanta@nxp.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>