Currently, a large part of the probe function runs before Runtime PM is
enabled.  As the driver manages the device's clock manually, this may
work fine on some systems, but may break on platforms with a more
complex power hierarchy.

Fix this by moving the initialization of Runtime PM before the first
register access (in cc_wait_for_reset_completion()), and putting the
device to sleep only after the last access (in cc_set_ree_fips_status()).

This allows to remove the pm_on flag, which was used to track manually
if Runtime PM had been enabled or not.
Remove the cc_pm_{init,go,fini}() wrappers, as they are called only
once, and obscure operation.
Signed-off-by: NGeert Uytterhoeven <geert+renesas@glider.be>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

SRAM addresses are small integer offsets into local SRAM.  Currently
they are stored using a mixture of cc_sram_addr_t (u64), u32, and
dma_addr_t types.

Settle on u32, and remove the cc_sram_addr_t typedefs.
This allows to drop several casts.
Signed-off-by: NGeert Uytterhoeven <geert+renesas@glider.be>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

The SRAM allocator does not support deallocating memory.
Hence remove all references to freeing SRAM.

Fix grammar while at it.
Signed-off-by: NGeert Uytterhoeven <geert+renesas@glider.be>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

While the larval digest addresses are not always used in
cc_get_plain_hmac_key() and cc_hash_digest(), they are always
calculated.

Defer their calculations to the points where needed.
Signed-off-by: NGeert Uytterhoeven <geert+renesas@glider.be>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Use the existing lower_32_bits() and upper_32_bits() macros instead of
explicit casts and shifts to split a 64-bit address in its two 32-bit
parts.
Drop the superfluous cast to "u16", as the FIELD_PREP() macro already
masks it to the specified field width.
Signed-off-by: NGeert Uytterhoeven <geert+renesas@glider.be>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

mlli_params.mlli_virt_addr is just a buffer of memory.
This allows to drop a cast.

No change in generated code.
Signed-off-by: NGeert Uytterhoeven <geert+renesas@glider.be>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Use devm_clk_get_optional() instead of devm_clk_get() and explicit
optional clock handling.
As clk_prepare_enable() and clk_disable_unprepare() handle optional
clocks fine, the cc_clk_on() and cc_clk_off() wrappers can be removed.

While at it, use the new "%pe" format specifier to print error codes.
Signed-off-by: NGeert Uytterhoeven <geert+renesas@glider.be>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

cc_sram_mgr_fini() doesn't do anything, so it can just be removed.
Signed-off-by: NGeert Uytterhoeven <geert+renesas@glider.be>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

When no SRAM can be allocated, cc_sram_alloc() already prints an error
message.  Hence there is no need to duplicate this in all callers.
Signed-off-by: NGeert Uytterhoeven <geert+renesas@glider.be>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Due to the way the hardware works, every double word in the SHA384 and
SHA512 larval hashes must be swapped.  Currently this is done at run
time, during driver initialization.

However, this swapping can easily be done at build time.  Treating each
double word as two words has the benefit of changing the larval hashes'
types from u64[] to u32[], like for all other hashes, and allows
dropping the casts and size doublings when calling cc_set_sram_desc().
Signed-off-by: NGeert Uytterhoeven <geert+renesas@glider.be>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Unneeded casts prevent the compiler from performing valuable checks.
This is especially true for function pointers.

Remove these casts, to prevent silently introducing bugs when a
variable's type might be changed in the future.

No change in generated code.
Signed-off-by: NGeert Uytterhoeven <geert+renesas@glider.be>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

If cc_queues_status() indicates that the queue is full,
cc_send_sync_request() should loop and retry.

However, cc_queues_status() returns either 0 (for success), or -ENOSPC
(for queue full), while cc_send_sync_request() checks for real errors by
comparing with -EAGAIN.  Hence -ENOSPC is always considered a real
error, and the code never retries the operation.

Fix this by just removing the check, as cc_queues_status() never returns
any other error value than -ENOSPC.
Signed-off-by: NGeert Uytterhoeven <geert+renesas@glider.be>
Acked-by: NGilad Ben-Yossef <gilad@benyossef.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Reading the debugfs files under /sys/kernel/debug/ccree/ can be done by
the user at any time.  On R-Car SoCs, the CCREE device is power-managed
using a moduile clock, and if this clock is not running, bogus register
values may be read.

Fix this by filling in the debugfs_regset32.dev field, so debugfs will
make sure the device is resumed while its registers are being read.

This fixes the bogus values (0x00000260) in the register dumps on R-Car
H3 ES1.0:

    -e6601000.crypto/regs:HOST_IRR = 0x00000260
    -e6601000.crypto/regs:HOST_POWER_DOWN_EN = 0x00000260
    +e6601000.crypto/regs:HOST_IRR = 0x00000038
    +e6601000.crypto/regs:HOST_POWER_DOWN_EN = 0x00000038
     e6601000.crypto/regs:AXIM_MON_ERR = 0x00000000
     e6601000.crypto/regs:DSCRPTR_QUEUE_CONTENT = 0x000002aa
    -e6601000.crypto/regs:HOST_IMR = 0x00000260
    +e6601000.crypto/regs:HOST_IMR = 0x017ffeff
     e6601000.crypto/regs:AXIM_CFG = 0x001f0007
     e6601000.crypto/regs:AXIM_CACHE_PARAMS = 0x00000000
    -e6601000.crypto/regs:GPR_HOST = 0x00000260
    +e6601000.crypto/regs:GPR_HOST = 0x017ffeff
     e6601000.crypto/regs:AXIM_MON_COMP = 0x00000000
    -e6601000.crypto/version:SIGNATURE = 0x00000260
    -e6601000.crypto/version:VERSION = 0x00000260
    +e6601000.crypto/version:SIGNATURE = 0xdcc63000
    +e6601000.crypto/version:VERSION = 0xaf400001

Note that this behavior is system-dependent, and the issue does not show
up on all R-Car Gen3 SoCs and boards.  Even when the device is
suspended, the module clock may be left enabled, if configured by the
firmware for Secure Mode, or when controlled by the Real-Time Core.
Signed-off-by: NGeert Uytterhoeven <geert+renesas@glider.be>
Acked-by: NGilad Ben-Yossef <gilad@benyossef.com>
Acked-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>
Reviewed-by: NNiklas Söderlund <niklas.soderlund@ragnatech.se>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Register qm to uacce framework for user crypto driver
Reviewed-by: NGreg Kroah-Hartman <gregkh@linuxfoundation.org>
Reviewed-by: NJonathan Cameron <Jonathan.Cameron@huawei.com>
Signed-off-by: NZhangfei Gao <zhangfei.gao@linaro.org>
Signed-off-by: NZhou Wang <wangzhou1@hisilicon.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Remove the module_param uacce_mode, which is not used currently.
Reviewed-by: NGreg Kroah-Hartman <gregkh@linuxfoundation.org>
Reviewed-by: NJonathan Cameron <Jonathan.Cameron@huawei.com>
Signed-off-by: NZhangfei Gao <zhangfei.gao@linaro.org>
Signed-off-by: NZhou Wang <wangzhou1@hisilicon.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

chtls_uld_add allocates room for info->nports net_device structs
following the chtls_dev struct, presumably because it was originally
intended that the ports array would be stored there. This is suggested
by the assignment which was present in initial versions and removed by
c4e84858 ("crypto: chelsio - remove redundant assignment to
cdev->ports"):

	cdev->ports = (struct net_device **)(cdev + 1);

This assignment was never used, being overwritten by lldi->ports
immediately afterwards, and I couldn't find any uses of the memory
allocated past the end of the struct.
Signed-off-by: NStephen Kitt <steve@sk2.org>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

QCE hangs when presented with an AES-XTS request whose length is larger
than QCE_SECTOR_SIZE (512-bytes), and is not a multiple of it.  Let the
fallback cipher handle them.
Signed-off-by: NEneas U de Queiroz <cotequeiroz@gmail.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Process small blocks using the fallback cipher, as a workaround for an
observed failure (DMA-related, apparently) when computing the GCM ghash
key.  This brings a speed gain as well, since it avoids the latency of
using the hardware engine to process small blocks.

Using software for all 16-byte requests would be enough to make GCM
work, but to increase performance, a larger threshold would be better.
Measuring the performance of supported ciphers with openssl speed,
software matches hardware at around 768-1024 bytes.

Considering the 256-bit ciphers, software is 2-3 times faster than qce
at 256-bytes, 30% faster at 512, and about even at 768-bytes.  With
128-bit keys, the break-even point would be around 1024-bytes.

This adds the 'aes_sw_max_len' parameter, to set the largest request
length processed by the software fallback.  Its default is being set to
512 bytes, a little lower than the break-even point, to balance the cost
in CPU usage.
Signed-off-by: NEneas U de Queiroz <cotequeiroz@gmail.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

The qce crypto driver appends an extra entry to the dst sgl, to maintain
private state information.

When the gcm driver sends requests to the ctr skcipher, it passes the
authentication tag after the actual crypto payload, but it must not be
touched.

Commit 1336c2221bee ("crypto: qce - save a sg table slot for result
buf") limited the destination sgl to avoid overwriting the
authentication tag but it assumed the tag would be in a separate sgl
entry.

This is not always the case, so it is better to limit the length of the
destination buffer to req->cryptlen before appending the result buf.
Signed-off-by: NEneas U de Queiroz <cotequeiroz@gmail.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Function dev_err() after platform_get_irq() is redundant because
platform_get_irq() already prints an error.
Signed-off-by: NChen Zhou <chenzhou10@huawei.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

No logs are recorded in dmesg during chcr module load, hence
adding the print and also appending -ko to driver version.
Signed-off-by: NDevulapally Shiva Krishna <shiva@chelsio.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

The libkcapi test which causes kernel panic is
aead asynchronous vmsplice multiple test.

./bin/kcapi  -v -d 4 -x 10   -c "ccm(aes)"
-q 4edb58e8d5eb6bc711c43a6f3693daebde2e5524f1b55297abb29f003236e43d
-t a7877c99 -n 674742abd0f5ba -k 2861fd0253705d7875c95ba8a53171b4
-a fb7bc304a3909e66e2e0c5ef952712dd884ce3e7324171369f2c5db1adc48c7d

This patch avoids dma_mapping of a zero length sg which causes the panic,
by using sg_nents_for_len which maps only upto a specific length
Signed-off-by: NAyush Sawal <ayush.sawal@chelsio.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

The libkcapi "cbc(aes)" failed tests are
symmetric asynchronous cipher one shot multiple test,
symmetric asynchronous cipher stream multiple test,
Symmetric asynchronous cipher vmsplice multiple test

In this patch a wait_for_completion is added in the chcr_aes_encrypt function,
which completes when the response of comes from the hardware.
This adds serialization for encryption in cbc(aes) aio case.
Signed-off-by: NAyush Sawal <ayush.sawal@chelsio.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Fixes gcc '-Wunused-but-set-variable' warning:

drivers/crypto/ccree/cc_cipher.c: In function 'cc_setup_state_desc':
drivers/crypto/ccree/cc_cipher.c:536:15: warning:
 variable 'du_size' set but not used [-Wunused-but-set-variable]

commit 5c83e8ec ("crypto: ccree - fix FDE descriptor sequence")
involved this unused variable, so remove it.
Reported-by: NHulk Robot <hulkci@huawei.com>
Signed-off-by: NYueHaibing <yuehaibing@huawei.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Remove the auth tag size from cryptlen before mapping the destination
in out-of-place AEAD decryption thus resolving a crash with
extended testmgr tests.
Signed-off-by: NGilad Ben-Yossef <gilad@benyossef.com>
Reported-by: NGeert Uytterhoeven <geert+renesas@glider.be>
Cc: stable@vger.kernel.org # v4.19+
Tested-by: NGeert Uytterhoeven <geert+renesas@glider.be>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Add reference counter incremented for each frame enqueued in CAAM
and replace unconditional sleep in empty_caam_fq() with polling the
reference counter.

When CONFIG_CRYPTO_MANAGER_EXTRA_TESTS=y boot time on LS1043A
platform with this optimization decreases from ~1100s to ~11s.
Signed-off-by: NValentin Ciocoi Radulescu <valentin.ciocoi@nxp.com>
Reviewed-by: NHoria Geantă <horia.geanta@nxp.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Fix an error causing no block sizes to be reported during
all AEAD registrations.
Signed-off-by: NGilad Ben-Yossef <gilad@benyossef.com>
Tested-by: NGeert Uytterhoeven <geert+renesas@glider.be>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

OFB and CTR modes block sizes were wrongfully reported as
the underlying block sizes. Fix it to 1 bytes as they
turn the block ciphers into stream ciphers.

Also document why our XTS differes from the generic
implementation.
Signed-off-by: NGilad Ben-Yossef <gilad@benyossef.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Make sure to only add the size of the auth tag to the source mapping
for encryption if it is an in-place operation. Failing to do this
previously caused us to try and map auth size len bytes from a NULL
mapping and crashing if both the cryptlen and assoclen are zero.
Reported-by: NGeert Uytterhoeven <geert+renesas@glider.be>
Tested-by: NGeert Uytterhoeven <geert+renesas@glider.be>
Signed-off-by: NGilad Ben-Yossef <gilad@benyossef.com>
Cc: stable@vger.kernel.org # v4.19+
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Deal gracefully with a NULL or empty scatterlist which can happen
if both cryptlen and assoclen are zero and we're doing in-place
AEAD encryption.

This fixes a crash when this causes us to try and map a NULL page,
at least with some platforms / DMA mapping configs.

Cc: stable@vger.kernel.org # v4.19+
Reported-by: NGeert Uytterhoeven <geert+renesas@glider.be>
Tested-by: NGeert Uytterhoeven <geert+renesas@glider.be>
Signed-off-by: NGilad Ben-Yossef <gilad@benyossef.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

If all possible errors occurs at the same time, the error_status will be
all 1s. The doorbell timeout error and FIFO overflow error will be print
in each cycle, which should be print just once.
Signed-off-by: NShukun Tan <tanshukun1@huawei.com>
Reviewed-by: NZhou Wang <wangzhou1@hisilicon.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

In error detect process, a lot of duplicate code can put into qm. We add
two callback(get_dev_hw_err_status and log_dev_hw_err) into struct
hisi_qm_err_ini to handle device error detect, meanwhile the qm error
detect not changed.
Signed-off-by: NShukun Tan <tanshukun1@huawei.com>
Signed-off-by: NZaibo Xu <xuzaibo@huawei.com>
Reviewed-by: NZhou Wang <wangzhou1@hisilicon.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Configure zip RAS error type in error handle initialization,
Where ECC 1bit is configured as CE error, others are NFE.
Signed-off-by: NShukun Tan <tanshukun1@huawei.com>
Reviewed-by: NZhou Wang <wangzhou1@hisilicon.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

The initialization and uninitialization of zip/hpre/sec/qm hardware error
is processed in respective drivers, which could be unified into qm.c. We
add struct hisi_qm_err_ini into struct hisi_qm, which involve all error
handlers of device and assignment should be done in driver probe.
Signed-off-by: NShukun Tan <tanshukun1@huawei.com>
Signed-off-by: NZaibo Xu <xuzaibo@huawei.com>
Reviewed-by: NZhou Wang <wangzhou1@hisilicon.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Add support for the crypto engine used in i.mx8mp (i.MX 8M "Plus"),
which is very similar to the one used in i.mx8mq, i.mx8mm, i.mx8mn.
Signed-off-by: NHoria Geantă <horia.geanta@nxp.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

There is a spelling mistake in an error message. Fix it.
Signed-off-by: NColin Ian King <colin.king@canonical.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

These inline versions of PM function for the case of CONFIG_PM is
not set are never used. Erase them.
Signed-off-by: NGilad Ben-Yossef <gilad@benyossef.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

cc_pm_put_suspend() return value was never checked and is not
useful. Turn it into a void functions.
Signed-off-by: NGilad Ben-Yossef <gilad@benyossef.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

We were using the irq field of the drvdata struct in
an overloaded fahsion - saving the IRQ number during init
and then storing the pending itnerrupt sources during
interrupt in the same field.

This worked because these usage are mutually exclusive but
are confusing. So simplify the code and change the init use
case to use a simple local variable.
Signed-off-by: NGilad Ben-Yossef <gilad@benyossef.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

The PM code was racy, possibly causing the driver to submit
requests to a powered down device. Fix the race and while
at it simplify the PM code.
Signed-off-by: NGilad Ben-Yossef <gilad@benyossef.com>
Fixes: 1358c13a ("crypto: ccree - fix resume race condition on init")
Cc: stable@kernel.org # v4.20
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>