If software fallback is used on older hardware accelerator setup (OMAP2/
OMAP3), the first block of data must be purged from the buffer. The
first block contains the pre-generated ipad value required by the HW,
but the software fallback algorithm generates its own, causing wrong
results.
Signed-off-by: NTero Kristo <t-kristo@ti.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

If we have processed any data with the hardware accelerator (digcnt > 0),
we must complete the entire hash by using it. This is because the current
hash value can't be imported to the software fallback algorithm. Otherwise
we end up with wrong hash results.
Signed-off-by: NTero Kristo <t-kristo@ti.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Some of the call paths of OMAP SHA driver can avoid executing the next
step of the crypto queue under tasklet; instead, execute the next step
directly via function call. This avoids a costly round-trip via the
scheduler giving a slight performance boost.
Signed-off-by: NTero Kristo <t-kristo@ti.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

The arm-neon-sha implementations have cra_priority of 150...300, so
increase omap-sham priority to 400 to ensure it is on top of any
software alg.
Signed-off-by: NBin Liu <b-liu@ti.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Adds software fallback support for small crypto requests. In these cases,
it is undesirable to use DMA, as setting it up itself is rather heavy
operation. Gives about 40% extra performance in ipsec usecase.
Signed-off-by: NBin Liu <b-liu@ti.com>
[t-kristo@ti.com: dropped the extra traces, updated some comments
 on the code]
Signed-off-by: NTero Kristo <t-kristo@ti.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

The extra call to dmaengine_terminate_all is not needed, as the DMA
is not running at this point. This improves performance slightly.
Signed-off-by: NLokesh Vutla <lokeshvutla@ti.com>
Signed-off-by: NTero Kristo <t-kristo@ti.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Change crypto queue size from 1 to 10 for omap SHA driver. This should
allow clients to enqueue requests more effectively to avoid serializing
whole crypto sequences, giving extra performance.
Signed-off-by: NTero Kristo <t-kristo@ti.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Calling runtime PM API for every block causes serious performance hit to
crypto operations that are done on a long buffer. As crypto is performed
on a page boundary, encrypting large buffers can cause a series of crypto
operations divided by page. The runtime PM API is also called those many
times.

Convert the driver to use runtime_pm autosuspend instead, with a default
timeout value of 1 second. This results in upto ~50% speedup.
Signed-off-by: NTero Kristo <t-kristo@ti.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

This if statement is reversed so we end up either leaking or Oopsing on
error.

Fixes: dbe24620 ('crypto: omap-sham - Use dma_request_chan() for requesting DMA channel')
Signed-off-by: NDan Carpenter <dan.carpenter@oracle.com>
Acked-by: NPeter Ujfalusi <peter.ujfalusi@ti.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

With the new dma_request_chan() the client driver does not need to look for
the DMA resource and it does not need to pass filter_fn anymore.
By switching to the new API the driver can now support deferred probing
against DMA.
Signed-off-by: NPeter Ujfalusi <peter.ujfalusi@ti.com>
CC: Herbert Xu <herbert@gondor.apana.org.au>
CC: David S. Miller <davem@davemloft.net>
CC: Lokesh Vutla <lokeshvutla@ti.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Signed-off-by: NDan Williams <dan.j.williams@intel.com>
[hch: split from a larger patch by Dan]
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Acked-by: NHerbert Xu <herbert@gondor.apana.org.au>
Signed-off-by: NJens Axboe <axboe@fb.com>

omap3 support is same as omap2, just with different IO address (specified in DT)
Signed-off-by: NPali Rohár <pali.rohar@gmail.com>
Acked-by: NPavel Machek <pavel@ucw.cz>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Function pm_runtime_get_sync could fail and we need to check return
value to prevent kernel crash.
Signed-off-by: NPali Rohár <pali.rohar@gmail.com>
Acked-by: NPavel Machek <pavel@ucw.cz>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

kmap_atomic() gives only the page address of the input page.
Driver should take care of adding the offset of the scatterlist
within the page to the returned page address.
omap-sham driver is not adding the offset to page and directly operates
on the return vale of kmap_atomic(), because of which the following
error comes when running crypto tests:

00000000: d9 a1 1b 7c aa 90 3b aa 11 ab cb 25 00 b8 ac bf
[    2.338169] 00000010: c1 39 cd ff 48 d0 a8 e2 2b fa 33 a1
[    2.344008] alg: hash: Chunking test 1 failed for omap-sha256

So adding the scatterlist offset to vaddr.
Signed-off-by: NLokesh Vutla <lokeshvutla@ti.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

omap_sham_handle_queue() can be called as part of done_task tasklet.
During this its atomic and any calls to pm functions cannot sleep.

But there is a call to pm_runtime_get_sync() (which can sleep) in
omap_sham_handle_queue(), because of which the following appears:
" [  116.169969] BUG: scheduling while atomic: kworker/0:2/2676/0x00000100"

Add pm_runtime_irq_safe() to avoid this.
Signed-off-by: NLokesh Vutla <lokeshvutla@ti.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

A platform_driver does not need to set an owner, it will be populated by the
driver core.
Signed-off-by: NWolfram Sang <wsa@the-dreams.de>

Replaced the use of a Variable Length Array In Struct (VLAIS) with a C99
compliant equivalent. This patch allocates the appropriate amount of memory
using a char array using the SHASH_DESC_ON_STACK macro.

The new code can be compiled with both gcc and clang.
Signed-off-by: NBehan Webster <behanw@converseincode.com>
Reviewed-by: NMark Charlebois <charlebm@gmail.com>
Reviewed-by: NJan-Simon Möller <dl9pf@gmx.de>
Acked-by: NHerbert Xu <herbert@gondor.apana.org.au>

HIGHMEM pages may not be mapped so we must kmap them before accessing.
This resolves a random OOPs error that was showing up during OpenSSL SHA tests.
Signed-off-by: NJoel Fernandes <joelf@ti.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Use SIMPLE_DEV_PM_OPS macro in order to make the code simpler.
Signed-off-by: NJingoo Han <jg1.han@samsung.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Command "tcrypt sec=1 mode=403" give the follwoing error for Polling
mode:
root@am335x-evm:/# insmod tcrypt.ko sec=1 mode=403
[...]

[  346.982754] test 15 ( 4096 byte blocks, 1024 bytes per update,   4 updates):   4352 opers/sec,  17825792 bytes/sec
[  347.992661] test 16 ( 4096 byte blocks, 4096 bytes per update,   1 updates):   7095 opers/sec,  29061120 bytes/sec
[  349.002667] test 17 ( 8192 byte blocks,   16 bytes per update, 512 updates):
[  349.010882] Unable to handle kernel NULL pointer dereference at virtual address 00000000
[  349.020037] pgd = ddeac000
[  349.022884] [00000000] *pgd=9dcb4831, *pte=00000000, *ppte=00000000
[  349.029816] Internal error: Oops: 17 [#1] PREEMPT SMP ARM
[  349.035482] Modules linked in: tcrypt(+)
[  349.039617] CPU: 0 PID: 1473 Comm: insmod Not tainted 3.12.4-01566-g6279006-dirty #38
[  349.047832] task: dda91540 ti: ddcd2000 task.ti: ddcd2000
[  349.053517] PC is at omap_sham_xmit_dma+0x6c/0x238
[  349.058544] LR is at omap_sham_xmit_dma+0x38/0x238
[  349.063570] pc : [<c04eb7cc>]    lr : [<c04eb798>]    psr: 20000013
[  349.063570] sp : ddcd3c78  ip : 00000000  fp : 9d8980b8
[  349.075610] r10: 00000000  r9 : 00000000  r8 : 00000000
[  349.081090] r7 : 00001000  r6 : dd898000  r5 : 00000040  r4 : ddb10550
[  349.087935] r3 : 00000004  r2 : 00000010  r1 : 53100080  r0 : 00000000
[  349.094783] Flags: nzCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
[  349.102268] Control: 10c5387d  Table: 9deac019  DAC: 00000015
[  349.108294] Process insmod (pid: 1473, stack limit = 0xddcd2248)

[...]

This is because polling_mode is not enabled for ctx without FLAGS_FINUP.

For polling mode the bufcnt is made 0 unconditionally. But it should be made 0
only if it is a final update or a total is not zero(This condition is similar
to what is done in DMA case). Because of this wrong hashes are produced.

Fixing the same.
Signed-off-by: NLokesh Vutla <lokeshvutla@ti.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

In omap_sham_probe() and omap_sham_remove(), 'dd->dma_lch'
is released without checking to see if it was successfully
requested or not.  This is a bug and was identified and
reported by Dan Carpenter here:

	http://www.spinics.net/lists/devicetree/msg11023.html

Add code to only release 'dd->dma_lch' when its not NULL
(that is, when it was successfully requested).
Reported-by: NDan Carpenter <dan.carpenter@oracle.com>
CC: Joel Fernandes <joelf@ti.com>
Signed-off-by: NMark A. Greer <mgreer@animalcreek.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Signed-off-by: NJoni Lapilainen <joni.lapilainen@gmail.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Replace some instances of of_irq_map_one()/irq_create_of_mapping() and
of_irq_to_resource() by the simpler equivalent irq_of_parse_and_map().
Signed-off-by: NThierry Reding <treding@nvidia.com>
Acked-by: NRob Herring <rob.herring@calxeda.com>
[grant.likely: resolved conflicts with core code renames]
Signed-off-by: NGrant Likely <grant.likely@linaro.org>

Each cycle of SHA512 operates on 32 data words where as
SHA256 operates on 16 data words. This needs to be updated
while configuring DMA channels. Doing the same.
Signed-off-by: NLokesh Vutla <lokeshvutla@ti.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

For writing input buffer into DATA_IN register current driver
has the following state machine:
-> if input buffer < 9 : use fallback driver
-> else if input buffer < block size : Copy input buffer into data_in regs
-> else use dma transfer.

In cases where requesting for DMA channels fails for some reason,
or channel numbers are not provided in DT or platform data, probe
also fails. Instead of returning from driver use cpu polling mode.
In this mode processor polls on INPUT_READY bit and writes data into
data_in regs when it equals 1. This operation is repeated until the
length of message.

Now the state machine looks like:
-> if input buffer < 9 : use fallback driver
-> else if input buffer < block size : Copy input buffer into data_in regs
-> else if dma enabled: use dma transfer
	   else use cpu polling mode.
Signed-off-by: NLokesh Vutla <lokeshvutla@ti.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Use devm_kzalloc() to make cleanup paths simpler.
Signed-off-by: NLokesh Vutla <lokeshvutla@ti.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Using devm_request_irq() rather than request_irq().
So removing free_irq() calls from the probe error
path and the remove handler.
Signed-off-by: NLokesh Vutla <lokeshvutla@ti.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Add support for the OMAP5 version of the SHAM module
that is present on OMAP5 and AM43xx SoCs.

This module is very simialar to OMAP4 version of SHAM module,
and adds SHA384 SHA512 hardware-accelerated hash functions to it.
To handle the higher digest size of SHA512, few SHA512_DIGEST_i
(i=1-16, and first 8 registers are duplicated from SHA_DIGEST_i
registers) registers are added at the end of register set.
So adding the above register offsets and module info in pdata.
Signed-off-by: NLokesh Vutla <lokeshvutla@ti.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Adding support for SHA348 and SHA512 in addition to MD5, SHA1, SHA224
SHA256 that the omap sha module supports.

In order to add the support
- Removed hard coded register offsets and passing offsets from pdata
- Updating Flag offsets so that they can be used for SHA256 and SHA512
- Adding the algo info.
Signed-off-by: NLokesh Vutla <lokeshvutla@ti.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Replace calls to deprecated devm_request_and_ioremap by devm_ioremap_resource.

Found with coccicheck and this semantic patch:
 scripts/coccinelle/api/devm_request_and_ioremap.cocci.
Signed-off-by: NLaurent Navet <laurent.navet@gmail.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

module_platform_driver() makes the code simpler by eliminating boilerplate
code.
Signed-off-by: NSachin Kamat <sachin.kamat@linaro.org>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

After DMA is complete, the omap_sham_finish_req function is called as
a part of the done_task tasklet. During this its atomic and any calls
to pm functions should not assume they wont sleep.

The patch replaces a call to pm_runtime_put_sync (which can sleep) with
pm_runtime_put thus fixing a kernel panic observed on AM33xx SoC during
SHA operation.

Tested on an AM33xx SoC device (beaglebone board).
To reproduce the problem, used the tcrypt kernel module as:
modprobe tcrypt sec=2 mode=403
Signed-off-by: NJoel A Fernandes <joelagnel@ti.com>
Cc: David S. Miller <davem@davemloft.net>
Acked-by: NMark A. Greer <mgreer@animalcreek.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Fix the compile errors created by commit 2545e8d
(crypto: omap-sham - Add Device Tree Support)
when CONFIG_OF is not defined.  This includes
changing omap_sham_get_res_dev() to omap_sham_get_res_of()
and creating an empty version of omap_sham_of_match[].
Signed-off-by: NMark A. Greer <mgreer@animalcreek.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

We still need to fix up few places for multiplatform support,
but that can proceed separately. Fix the issue by making the
problem drivers depends !ARCH_MULTIPLATFORM for now.

The remaining pieces that are not multiplatform compatible
for omap2+ SoCs are:

1. Some drivers are using custom omap_dm_timer calls

There are two drivers that are directly usign omap hardware
timers for PWM and DSP clocking: drivers/media/rc/ir-rx51.c and
drivers/staging/tidspbridge/core/dsp-clock.c. These can be
fixed for multiplatform by allowing a minimal set of hardware
timers to be accessed, and for some functionality by using the
hrtimer framework.

2. Hardware OMAP4_ERRATA_I688 needs to be fixed up

This can't be enabled for multiplatform configurations in
it's current form. It may be possible to fix it up to do
instruction replacement early on during init. Luckily it
looks like this errata does not seem to get hit with
mainline kernel code alone at least currently.

3. Legacy header needed for omap-sham.c

Looks like it still needs mach/irqs.h for omap1 that
does not exist for multiplatform systems. Just ifdef
it for now.

4. Mailbox is waiting to get moved to drivers

Disable it for now to avoid adding a dependency to the
mailbox patches.

Cc: Timo Kokkonen <timo.t.kokkonen@iki.fi>
Cc: Sean Young <sean@mess.org>
Cc: "Víctor Manuel Jáquez Leal" <vjaquez@igalia.com>
Cc: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Cc: Mauro Carvalho Chehab <mchehab@redhat.com>
Cc: Omar Ramirez Luna <omar.ramirez@ti.com>
Cc: Herbert Xu <herbert@gondor.apana.org.au>
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Cc: Santosh Shilimkar <santosh.shilimkar@ti.com>
Tested-by: NEzequiel Garcia <ezequiel.garcia@free-electrons.com>
[tony@atomide.com: updated to disable mailbox]
Signed-off-by: NTony Lindgren <tony@atomide.com>

The OMAP4/AM33xx version of the SHAM crypto module
supports SHA224 and SHA256 in addition to MD5 and
SHA1 that the OMAP2 version of the module supports.

To add this support, use the platform_data introduced
in an ealier commit to hold the list of algorithms
supported by the current module.  The probe routine
will use that list to register the correct algorithms.

Note: The code being integrated is from the TI AM33xx SDK
and was written by Greg Turner <gkmturner@gmail.com> and
Herman Schuurman (current email unknown) while at TI.

CC: Greg Turner <gkmturner@gmail.com>
CC: Dmitry Kasatkin <dmitry.kasatkin@intel.com>
Signed-off-by: NMark A. Greer <mgreer@animalcreek.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Add support for the OMAP4 version of the SHAM module
that is present on OMAP4 and AM33xx SoCs.

The modules have several differences including register
offsets, hardware XORing, and how DMA is triggered.
To handle these differences, a platform_data structure
is defined and contains routine pointers, register offsets,
bit shifts within registers, and flags to indicate whether
the hardware supports XORing and provides SHA1 results in
big or little endian.  OMAP2/OMAP3-specific routines are
suffixed with '_omap2' and OMAP4/AM33xx routines are suffixed
with '_omap4'.

Note: The code being integrated is from the TI AM33xx SDK
and was written by Greg Turner <gkmturner@gmail.com> and
Herman Schuurman (current email unknown) while at TI.

CC: Greg Turner <gkmturner@gmail.com>
CC: Dmitry Kasatkin <dmitry.kasatkin@intel.com>
Signed-off-by: NMark A. Greer <mgreer@animalcreek.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Use the dma_request_slave_channel_compat() call instead of
the dma_request_channel() call to request a DMA channel.
This allows the omap-sham driver use different DMA engines.

CC: Dmitry Kasatkin <dmitry.kasatkin@intel.com>
Signed-off-by: NMark A. Greer <mgreer@animalcreek.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Add Device Tree suport to the omap-sham crypto
driver.  Currently, only support for OMAP2 and
OMAP3 is being added but support for OMAP4 will
be added in a subsequent patch.

CC: Dmitry Kasatkin <dmitry.kasatkin@intel.com>
Signed-off-by: NMark A. Greer <mgreer@animalcreek.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Remove usage of the private OMAP DMA API.
The dmaengine API will be used instead.

CC: Russell King <rmk+kernel@arm.linux.org.uk>
CC: Dmitry Kasatkin <dmitry.kasatkin@intel.com>
Signed-off-by: NMark A. Greer <mgreer@animalcreek.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Add code to use the new dmaengine API alongside
the existing DMA code that uses the private
OMAP DMA API.  The API to use is chosen by
defining or undefining 'OMAP_SHAM_DMA_PRIVATE'.

This is a transitional change and the code that uses
the private DMA API will be removed in an upcoming
commit.

CC: Russell King <rmk+kernel@arm.linux.org.uk>
CC: Dmitry Kasatkin <dmitry.kasatkin@intel.com>
Signed-off-by: NMark A. Greer <mgreer@animalcreek.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>