With the new dma_request_chan() the client driver does not need to look for
the DMA resource and it does not need to pass filter_fn anymore.
By switching to the new API the driver can now support deferred probing
against DMA.
Signed-off-by: NPeter Ujfalusi <peter.ujfalusi@ti.com>
CC: Herbert Xu <herbert@gondor.apana.org.au>
CC: David S. Miller <davem@davemloft.net>
CC: Lokesh Vutla <lokeshvutla@ti.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Since the crypto engine framework had been merged, thus this patch integrates
with the newly added crypto engine framework to make the crypto hardware
engine under utilized as each block needs to be processed before the crypto
hardware can start working on the next block.

The crypto engine framework can manage and process the requests automatically,
so remove the 'queue' and 'queue_task' things in omap des driver.
Signed-off-by: NBaolin <baolin.wang@linaro.org>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Removed unused variable "err" and directly return "0"

Reported by coccicheck -
./drivers/crypto/omap-aes.c:542:5-8: Unneeded variable: "err". Return "0" on line 551
./drivers/crypto/omap-des.c:530:5-8: Unneeded variable: "err". Return "0" on line 539
Signed-off-by: NRahul Pathak <rpathak@visteon.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

When using DES module the next bug appears:

    BUG: scheduling while atomic: kworker/0:1/63/0x00000102

With backtrace as follows:

<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< cut here >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

[<c0012294>] (dump_backtrace) from [<c00124ac>] (show_stack+0x18/0x1c)
[<c0012494>] (show_stack) from [<c0752554>] (dump_stack+0x84/0xc4)
[<c07524d0>] (dump_stack) from [<c0750218>] (__schedule_bug+0x54/0x64)
[<c07501c4>] (__schedule_bug) from [<c07548a4>] (__schedule+0x4ac/0x53c)
[<c07543f8>] (__schedule) from [<c075496c>] (schedule+0x38/0x88)
[<c0754934>] (schedule) from [<c03c3984>] (rpm_resume+0x158/0x59c)
[<c03c382c>] (rpm_resume) from [<c03c3e1c>] (__pm_runtime_resume+0x54/0x6c)
[<c03c3dc8>] (__pm_runtime_resume) from [<c0568ff8>] (omap_des_handle_queue+0x154/0x7bc)
[<c0568ea4>] (omap_des_handle_queue) from [<c05696b8>] (omap_des_crypt+0x58/0xbc)
[<c0569660>] (omap_des_crypt) from [<c0569730>] (omap_des_cbc_decrypt+0x14/0x18)
[<c056971c>] (omap_des_cbc_decrypt) from [<c0297534>] (authenc_verify_ahash_done+0xe0/0xe8)
[<c0297454>] (authenc_verify_ahash_done) from [<c056a330>] (omap_sham_finish_req+0x58/0xa8)
[<c056a2d8>] (omap_sham_finish_req) from [<c056b714>] (omap_sham_done_task+0x1c0/0x1e0)
[<c056b554>] (omap_sham_done_task) from [<c003e53c>] (tasklet_action+0x80/0x118)
[<c003e4bc>] (tasklet_action) from [<c003e740>] (__do_softirq+0x11c/0x260)
[<c003e624>] (__do_softirq) from [<c003eb64>] (irq_exit+0xc0/0xfc)
[<c003eaa4>] (irq_exit) from [<c000f1c4>] (handle_IRQ+0x4c/0x98)
[<c000f178>] (handle_IRQ) from [<c0008568>] (gic_handle_irq+0x34/0x64)
[<c0008534>] (gic_handle_irq) from [<c0758540>] (__irq_svc+0x40/0x70)

<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< cut here >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

Insight was seen in drivers/crypto/omap-sham.c driver.
All credits for this patch go to Grygorii Strashko.
Signed-off-by: NSam Protsenko <semen.protsenko@linaro.org>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

dma_unmap_sg() is being called twice after completing the
task. Looks like this is a copy paste error when creating
des driver.
With this the following warn appears during boot:

[    4.210457] ------------[ cut here ]------------
[    4.215114] WARNING: CPU: 0 PID: 0 at lib/dma-debug.c:1080 check_unmap+0x710/0x9a0()
[    4.222899] omap-des 480a5000.des: DMA-API: device driver tries to free DMA memory it has not allocated [device address=0x00000000ab2ce000] [size=8 bytes]
[    4.236785] Modules linked in:
[    4.239860] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.14.39-02999-g1bc045a-dirty #182
[    4.247918] [<c001678c>] (unwind_backtrace) from [<c0012574>] (show_stack+0x10/0x14)
[    4.255710] [<c0012574>] (show_stack) from [<c05a37e8>] (dump_stack+0x84/0xb8)
[    4.262977] [<c05a37e8>] (dump_stack) from [<c0046464>] (warn_slowpath_common+0x68/0x8c)
[    4.271107] [<c0046464>] (warn_slowpath_common) from [<c004651c>] (warn_slowpath_fmt+0x30/0x40)
[    4.279854] [<c004651c>] (warn_slowpath_fmt) from [<c02d50a4>] (check_unmap+0x710/0x9a0)
[    4.287991] [<c02d50a4>] (check_unmap) from [<c02d5478>] (debug_dma_unmap_sg+0x90/0x19c)
[    4.296128] [<c02d5478>] (debug_dma_unmap_sg) from [<c04a77d8>] (omap_des_done_task+0x1cc/0x3e4)
[    4.304963] [<c04a77d8>] (omap_des_done_task) from [<c004a090>] (tasklet_action+0x84/0x124)
[    4.313370] [<c004a090>] (tasklet_action) from [<c004a4ac>] (__do_softirq+0xf0/0x20c)
[    4.321235] [<c004a4ac>] (__do_softirq) from [<c004a840>] (irq_exit+0x98/0xec)
[    4.328500] [<c004a840>] (irq_exit) from [<c000f9ac>] (handle_IRQ+0x50/0xb0)
[    4.335589] [<c000f9ac>] (handle_IRQ) from [<c0008688>] (gic_handle_irq+0x28/0x5c)

Removing the duplicate call to dma_unmap_sg().

Cc: stable@vger.kernel.org
Reported-by: NTomi Valkeinen <tomi.valkeinen@ti.com>
Signed-off-by: NLokesh Vutla <lokeshvutla@ti.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Modify crypto drivers to use the generic SG helper since
both of them are equivalent and the one from crypto is redundant.

See also:
  468577ab   reverted in
  b2ab4a57Signed-off-by: NCristian Stoica <cristian.stoica@freescale.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

dd->total is unsigned so it won't do any good to check for negative
value after subtracting instead of that we should check if the
subtracted value is bigger than him

This was partially found by using a static code analysis program
called cppcheck.
Signed-off-by: NAsaf Vertz <asaf.vertz@tandemg.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

A platform_driver does not need to set an owner, it will be populated by the
driver core.
Signed-off-by: NWolfram Sang <wsa@the-dreams.de>

pm_runtime_get_sync may not always succeed depending on SoC involved.
So handle the error appropriately ensuring usage_count is accurate in
case of failure.
Signed-off-by: NNishanth Menon <nm@ti.com>
Reviewed-by: NFelipe Balbi <balbi@ti.com>
Reported-by: NJoachim Eastwood <manabian@gmail.com>
Tested-by: NJoachim Eastwood <manabian@gmail.com>
Acked-by: NJoel Fernandes <joelf@ti.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Use devm_ioremap_resource() because devm_request_and_ioremap() is
obsoleted by devm_ioremap_resource().
Signed-off-by: NJingoo Han <jg1.han@samsung.com>
Acked-by: NJoel Fernandes <joelf@ti.com>
Reviewed-by: NMarek Vasut <marex@denx.de>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Make omap_des_copy_needed(), omap_des_copy_sgs(), because these
functions are used only in this file.
Signed-off-by: NJingoo Han <jg1.han@samsung.com>
Acked-by: NJoel Fernandes <joelf@ti.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Use SIMPLE_DEV_PM_OPS macro in order to make the code simpler.
Signed-off-by: NJingoo Han <jg1.han@samsung.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Add omap-des driver with platform data for OMAP4/AM43xx. Support added for DES
ECB and CBC modes. Also add support for 3DES operation where 3 64-bit keys are
used to perform a DES encrypt-decrypt-encrypt (des3_ede) operation on a buffer.

Tests have been conducted with the CRYPTO test manager, and functionality is
verified at different page length alignments.
Signed-off-by: NJoel Fernandes <joelf@ti.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

The AES driver currently assumes that pm_runtime_get_sync will always
succeed, which may not always be true, so add error handling for the
same.

This scenario was reported in the following bug:
place.  https://bugzilla.kernel.org/show_bug.cgi?id=66441Reported-by: NTobias Jakobi <tjakobi@math.uni-bielefeld.de>
Signed-off-by: NNishanth Menon <nm@ti.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

NIST vectors for CTR mode in testmgr.h assume the entire IV as the counter. To
get correct results that match the output of these vectors, we need to set the
counter length correctly.
Signed-off-by: NJoel Fernandes <joelf@ti.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Local symbols used only in this file are made static.
Signed-off-by: NSachin Kamat <sachin.kamat@linaro.org>
Cc: Dmitry Kasatkin <dmitry.kasatkin@nokia.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Keeps request_irq exit/error code paths simpler.
Suggested-by: NLokesh Vutla <lokeshvutla@ti.com>
Signed-off-by: NJoel Fernandes <joelf@ti.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Use devm_kzalloc instead of kzalloc. With this change, there is no need to
call kfree in error/exit paths.
Signed-off-by: NJoel Fernandes <joelf@ti.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

For cases where offset/length of on any page of the input SG is not aligned by
AES_BLOCK_SIZE, we copy all the pages from the input SG list into a contiguous
buffer and prepare a single element SG list for this buffer with length as the
total bytes to crypt.

This is requried for cases such as when an SG list of 16 bytes total size
contains 16 pages each containing 1 byte. DMA using the direct buffers of such
instances is not possible.

For this purpose, we first detect if the unaligned case and accordingly
allocate enough number of pages to satisfy the request and prepare SG lists.
We then copy data into the buffer, and copy data out of it on completion.
Signed-off-by: NJoel Fernandes <joelf@ti.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

In cases where requesting for DMA channels fails for some reason, or channel
numbers are not provided in DT or platform data, we switch to PIO-only mode
also checking if platform provides IRQ numbers and interrupt register offsets
in DT and platform data. All dma-only paths are avoided in this mode.
Signed-off-by: NJoel Fernandes <joelf@ti.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

We initialize the scatter gather walk lists needed for PIO mode and avoid all
DMA paths such as mapping/unmapping buffers by checking for the pio_only flag.
Signed-off-by: NJoel Fernandes <joelf@ti.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

We add an IRQ handler that implements a state-machine for PIO-mode and data
structures for walking the scatter-gather list. The IRQ handler is called in
succession both when data is available to read or next data can be sent for
processing. This process continues till the entire in/out SG lists have been
walked. Once the SG-list has been completely walked, the IRQ handler schedules
the done_task tasklet.

Also add a useful macro that is used through out the IRQ code for a common
pattern of calculating how much an SG list has been walked.  This improves code
readability and avoids checkpatch errors.
Signed-off-by: NJoel Fernandes <joelf@ti.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Add IRQ information to pdata and helper macros. These are required
for PIO-mode support.
Signed-off-by: NJoel Fernandes <joelf@ti.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Intermdiate buffers were allocated, mapped and used for DMA.  These are no
longer required as we use the SGs from crypto layer directly in previous
commits in the series. Also along with it, remove the logic for copying SGs
etc as they are no longer used, and all the associated variables in omap_aes_device.
Signed-off-by: NJoel Fernandes <joelf@ti.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Earlier functions that did a similar sync are replaced by the dma_sync_sg_*
which can operate on entire SG list.
Signed-off-by: NJoel Fernandes <joelf@ti.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

In early version of this driver, assumptions were made such as DMA layer
requires contiguous buffers etc. Due to this, new buffers were allocated,
mapped and used for DMA. These assumptions are no longer true and DMAEngine
scatter-gather DMA doesn't have such requirements. We simply the DMA operations
by directly using the scatter-gather buffers provided by the crypto layer
instead of creating our own.

Lot of logic that handled DMA'ing only X number of bytes of the total, or as
much as fitted into a 3rd party buffer is removed and is no longer required.

Also, good performance improvement of atleast ~20% seen with encrypting a
buffer size of 8K (1800 ops/sec vs 1400 ops/sec).  Improvement will be higher
for much larger blocks though such benchmarking is left as an exercise for the
reader.  Also DMA usage is much more simplified and coherent with rest of the
code.
Signed-off-by: NJoel Fernandes <joelf@ti.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Crypto layer only passes nbytes but number of SG elements is needed for mapping
or unmapping SGs at one time using dma_map* API and also needed to pass in for
dmaengine prep function.

We call function added to scatterwalk for this purpose in omap_aes_handle_queue
to populate the values which are used later.
Signed-off-by: NJoel Fernandes <joelf@ti.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

When DEBUG is enabled, these macros can be used to print variables in integer
and hex format, and clearly display which registers, offsets and values are
being read/written , including printing the names of the offsets and their values.

Using statement expression macros in read path as,
Suggested-by: NJoe Perches <joe@perches.com>
Signed-off-by: NJoel Fernandes <joelf@ti.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Calling runtime PM API for every block causes serious perf hit to
crypto operations that are done on a long buffer.
As crypto is performed on a page boundary, encrypting large buffers can
cause a series of crypto operations divided by page. The runtime PM API
is also called those many times.

We call runtime_pm_get_sync only at beginning on the session (cra_init)
and runtime_pm_put at the end. This result in upto a 50% speedup as below.
This doesn't make the driver to keep the system awake as runtime get/put
is only called during a crypto session which completes usually quickly.

Before:
root@beagleboard:~# time -v openssl speed -evp aes-128-cbc
Doing aes-128-cbc for 3s on 16 size blocks: 13310 aes-128-cbc's in 0.01s
Doing aes-128-cbc for 3s on 64 size blocks: 13040 aes-128-cbc's in 0.04s
Doing aes-128-cbc for 3s on 256 size blocks: 9134 aes-128-cbc's in 0.03s
Doing aes-128-cbc for 3s on 1024 size blocks: 8939 aes-128-cbc's in 0.01s
Doing aes-128-cbc for 3s on 8192 size blocks: 4299 aes-128-cbc's in 0.00s

After:
root@beagleboard:~# time -v openssl speed -evp aes-128-cbc
Doing aes-128-cbc for 3s on 16 size blocks: 18911 aes-128-cbc's in 0.02s
Doing aes-128-cbc for 3s on 64 size blocks: 18878 aes-128-cbc's in 0.02s
Doing aes-128-cbc for 3s on 256 size blocks: 11878 aes-128-cbc's in 0.10s
Doing aes-128-cbc for 3s on 1024 size blocks: 11538 aes-128-cbc's in 0.05s
Doing aes-128-cbc for 3s on 8192 size blocks: 4857 aes-128-cbc's in 0.03s

While at it, also drop enter and exit pr_debugs, in related code. tracers
can be used for that.

Tested on a Beaglebone (AM335x SoC) board.
Signed-off-by: NJoel A Fernandes <joelagnel@ti.com>
Acked-by: NKevin Hilman <khilman@linaro.org>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Replace calls to deprecated devm_request_and_ioremap by devm_ioremap_resource.

Found with coccicheck and this semantic patch:
 scripts/coccinelle/api/devm_request_and_ioremap.cocci.
Signed-off-by: NLaurent Navet <laurent.navet@gmail.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

module_platform_driver() makes the code simpler by eliminating boilerplate
code.
Signed-off-by: NSachin Kamat <sachin.kamat@linaro.org>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

After DMA is complete, the omap_aes_finish_req function is called as
a part of the done_task tasklet. During this its atomic and any calls
to pm functions should not assume they wont sleep.

The patch replaces a call to pm_runtime_put_sync (which can sleep) with
pm_runtime_put thus fixing a kernel panic observed on AM33xx SoC during
AES operation.

Tested on an AM33xx SoC device (beaglebone board).
To reproduce the problem, I used the tcrypt kernel module as:
modprobe tcrypt sec=2 mode=500
Signed-off-by: NJoel A Fernandes <joelagnel@ti.com>
Cc: David S. Miller <davem@davemloft.net>
Acked-by: NMark A. Greer <mgreer@animalcreek.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

The OMAP3 and OMAP4/AM33xx versions of the AES crypto
module support the CTR algorithm in addition to ECB
and CBC that the OMAP2 version of the module supports.

So, OMAP2 and OMAP3 share a common register set but
OMAP3 supports CTR while OMAP2 doesn't.  OMAP4/AM33XX
uses a different register set from OMAP2/OMAP3 and
also supports CTR.

To add this support, use the platform_data introduced
in an ealier commit to hold the list of algorithms
supported by the current module.  The probe routine
will use that list to register the correct algorithms.

Note: The code being integrated is from the TI AM33xx SDK
and was written by Greg Turner <gkmturner@gmail.com> and
Herman Schuurman (current email unknown) while at TI.

CC: Greg Turner <gkmturner@gmail.com>
CC: Dmitry Kasatkin <dmitry.kasatkin@intel.com>
Signed-off-by: NMark A. Greer <mgreer@animalcreek.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Add support for the OMAP4 version of the AES module
that is present on OMAP4 and AM33xx SoCs.

The modules have several differences including register
offsets and how DMA is triggered.  To handle these
differences, a platform_data structure is defined and
contains routine pointers, register offsets, and bit
offsets within registers.  OMAP2/OMAP3-specific routines
are suffixed with '_omap2' and OMAP4/AM33xx routines are
suffixed with '_omap4'.

Note: The code being integrated is from the TI AM33xx SDK
and was written by Greg Turner <gkmturner@gmail.com> and
Herman Schuurman (current email unknown) while at TI.

CC: Greg Turner <gkmturner@gmail.com>
CC: Dmitry Kasatkin <dmitry.kasatkin@intel.com>
Signed-off-by: NMark A. Greer <mgreer@animalcreek.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Use the dma_request_slave_channel_compat() call instead of
the dma_request_channel() call to request a DMA channel.
This allows the omap-aes driver use different DMA engines.

CC: Dmitry Kasatkin <dmitry.kasatkin@intel.com>
Signed-off-by: NMark A. Greer <mgreer@animalcreek.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Add Device Tree suport to the omap-aes crypto
driver.  Currently, only support for OMAP2 and
OMAP3 is being added but support for OMAP4 will
be added in a subsequent patch.

CC: Dmitry Kasatkin <dmitry.kasatkin@intel.com>
Signed-off-by: NMark A. Greer <mgreer@animalcreek.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Remove usage of the private OMAP DMA API.
The dmaengine API will be used instead.

CC: Russell King <rmk+kernel@arm.linux.org.uk>
CC: Dmitry Kasatkin <dmitry.kasatkin@intel.com>
Signed-off-by: NMark A. Greer <mgreer@animalcreek.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Add code to use the new dmaengine API alongside
the existing DMA code that uses the private
OMAP DMA API.  The API to use is chosen by
defining or undefining 'OMAP_AES_DMA_PRIVATE'.

CC: Russell King <rmk+kernel@arm.linux.org.uk>
CC: Dmitry Kasatkin <dmitry.kasatkin@intel.com>
Signed-off-by: NMark A. Greer <mgreer@animalcreek.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Add suspend/resume support to the OMAP AES driver.

CC: Dmitry Kasatkin <dmitry.kasatkin@intel.com>
Signed-off-by: NMark A. Greer <mgreer@animalcreek.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>

Convert the omap-aes crypto driver to use the
pm_runtime API instead of the clk API.

CC: Kevin Hilman <khilman@deeprootsystems.com>
CC: Dmitry Kasatkin <dmitry.kasatkin@intel.com>
Signed-off-by: NMark A. Greer <mgreer@animalcreek.com>
Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>