If the client queues up more transfers the driver will not able to move to
the next transfer without knowing that the previous descriptor is
completed.
Signed-off-by: NPeter Ujfalusi <peter.ujfalusi@ti.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

When based on the CCR_ENABLE bit the channel is stopped we should not call
omap_dma_callback(), only change the return value to DMA_COMPLETE. Client
drivers will do the right thing to clean up the channel after the transfer
has been completed.
Check the CCR_ENABLE only if the channel is running and not paused since
pause in sDMA means that the channel is stopped.
This will fix one hard to reproduce race condition when the channel is
terminated during transfer (affecting cyclic operation).

Fixes: 1a7cf7b2 ("dmaengine: omap-dma: Handle cases when the channel is polled for completion")
Signed-off-by: NPeter Ujfalusi <peter.ujfalusi@ti.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

We need the callback to support the dmaengine_terminate_sync().
Signed-off-by: NPeter Ujfalusi <peter.ujfalusi@ti.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

Add support for providing device to filter_fn mapping so client drivers
can switch to use the dma_request_chan() API.
Signed-off-by: NPeter Ujfalusi <peter.ujfalusi@ti.com>
Reviewed-by: NArnd Bergmann <arnd@arndb.de>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

When a DMA client driver decides that it is not providing callback for
completion of a transfer (and/or does not set the DMA_PREP_INTERRUPT) but
it will poll the status of the transfer (in case of short memcpy for
example) we will not get interrupt for the completion of the transfer and
will not mark the transaction as done.
Check the channel enable bit in the CCR when the status is queried and if
the channel is no longer active, we call the omap_dma_callback() to handle
the transfer completion.
Signed-off-by: NPeter Ujfalusi <peter.ujfalusi@ti.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

The use of tasklet to actually start the DMA transfer slightly decreases the
DMA throughput since it adds small scheduling delay when the transfer is
started. In normal use, even with high I/O load the tasklet would start
one transaction at a time, however running the DMAtest for memcpy on all
available channels will cause the tasklet to start about 15 transfers.
The performance numbers on OMAP4 PandaBoard-es (test_buf_size = 6553):
With the tasklet:
dmatest: dma0chan30-copy: summary 5000 tests, 0 failures 186 iops 593 KB/s (0)
dmatest: dma0chan8-copy0: summary 5000 tests, 0 failures 184 iops 584 KB/s (0)
dmatest: dma0chan13-copy: summary 5000 tests, 0 failures 184 iops 585 KB/s (0)
dmatest: dma0chan12-copy: summary 5000 tests, 0 failures 184 iops 585 KB/s (0)
dmatest: dma0chan7-copy0: summary 5000 tests, 0 failures 183 iops 581 KB/s (0)

With this patch (no tasklet):
dmatest: dma0chan4-copy0: summary 5000 tests, 0 failures 199 iops 644 KB/s (0)
dmatest: dma0chan5-copy0: summary 5000 tests, 0 failures 199 iops 645 KB/s (0)
dmatest: dma0chan6-copy0: summary 5000 tests, 0 failures 199 iops 637 KB/s (0)
dmatest: dma0chan24-copy: summary 5000 tests, 0 failures 199 iops 638 KB/s (0)
dmatest: dma0chan16-copy: summary 5000 tests, 0 failures 199 iops 638 KB/s (0)
Signed-off-by: NPeter Ujfalusi <peter.ujfalusi@ti.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

The for_each_sg() macro's last parameter is inteded to be used as counter.
We can use 'i' instead of 'j' within the loop for indexes.
Signed-off-by: NPeter Ujfalusi <peter.ujfalusi@ti.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

During mem copy both src and dst position moves at the same pace. Check the
dst position for progress reporting.
Signed-off-by: NPeter Ujfalusi <peter.ujfalusi@ti.com>
Tested-by: NTomi Valkeinen <tomi.valkeinen@ti.com>
Signed-off-by: NJyri Sarha <jsarha@ti.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

The L3 throughput can be higher than expected when packed access
is not enabled.  The ratio depends on the number of bytes in a
transaction and the EMIF interface width.

The throughput was measured for the following settings/cases:

* Case 1: Burst size of 64 bytes, packed access disabled
* Case 2: Burst size of 64 bytes, packed access enabled
* Case 3: Burst disabled, packed access disabled

Throughput measurements were done during McASP-based audio
playback on the Jacinto6 EVM using the omapconf tool [1]:
$ omapconf trace bw -m sdma_rd

 ---------------------------------------------------------
                                  Throughput (MB/s)
  Audio parameters            Case 1    Case 2    Case 3
 ---------------------------------------------------------
  44.1kHz, 16-bits, stereo      1.41      0.18      1.41
  44.1kHz, 32-bits, stereo      1.41      0.35      1.41
  44.1kHz, 16-bits, 4-chan      2.82      0.35      2.82
  44.1kHz, 16-bits, 6-chan      4.23      0.53      4.23
  44.1kHz, 16-bits, 8-chan      5.64      0.71      5.64
 ---------------------------------------------------------

From above measurements, case 2 is the only one that delivers
the expected throughput for the given audio parameters.  For
that reason, the packed accesses are now enabled.

It's worth to mention that packed accesses cannot be enabled
for all addressing modes. In cyclic transfers, it can be
enabled in the source for MEM_TO_DEV and in dest for DEV_TO_MEM,
as they use post-increment mode which supports packed accesses.

Peter Ujfalusi:
From the TRM regarding to this:
"NOTE: Except in the constant addressing mode, the source or
destination must be specified as packed for burst transactions
to occur."

So w/o the packed setting the burst on the MEM side was not
enabled, this explains the numbers.

[1] https://github.com/omapconf/omapconfSigned-off-by: NMisael Lopez Cruz <misael.lopez@ti.com>
Signed-off-by: NPeter Ujfalusi <peter.ujfalusi@ti.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

Since the mapping between the hardware request lines and channels has been
removed it no longer make sense to have too many channels.
Set the number of channels to match with the number of logical channels
supported by sDMA.
Signed-off-by: NPeter Ujfalusi <peter.ujfalusi@ti.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

Do not direct map the virtual channels to sDMA request number. When the
sDMA is behind of a crossbar this direct mapping can cause situations when
certain channel can not be requested since the crossbar request number
will no longer match with the sDMA request line.
The direct mapping for virtual channels with HW request lines will make it
harder to implement MEM_TO_MEM mode for the driver.
Signed-off-by: NPeter Ujfalusi <peter.ujfalusi@ti.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

Use the dma-requests property from DT to get the number of DMA requests.
In case of legacy boot or failure to find the property, use the default
127 as number of requests.
Signed-off-by: NPeter Ujfalusi <peter.ujfalusi@ti.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

Instead of magic numbers in the code, use define for number of logical DMA
channels and DMA requests.
Signed-off-by: NPeter Ujfalusi <peter.ujfalusi@ti.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

The sDMA controller is capable of performing memory copy operation. It need
to be configured to software triggered mode and without HW synchronization.
The sDMA can copy data which is aligned to 8, 16 or 32 bits.
Signed-off-by: NPeter Ujfalusi <peter.ujfalusi@ti.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

In omap_dma_start_desc the vdesc->node is removed from the virt-dma
framework managed lists (to be precise from the desc_issued list).
If a terminate_all comes before the transfer finishes the omap_desc will
not be freed up because it is not in any of the lists and we stopped the
DMA channel so the transfer will not going to complete.
There is no special sequence for leaking memory when using cyclic (audio)
transfer: with every start and stop of a cyclic transfer the driver leaks
struct omap_desc worth of memory.

Free up the allocated memory directly in omap_dma_terminate_all() since the
framework will not going to do that for us.
Signed-off-by: NPeter Ujfalusi <peter.ujfalusi@ti.com>
CC: <stable@vger.kernel.org>
CC: <linux-omap@vger.kernel.org>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

Signed-off-by: NVinod Koul <vinod.koul@intel.com>

Now that the generic slave caps code can make use of the device assigned
capabilities, instead of relying on a callback to be implemented.

Make use of this code.
Signed-off-by: NMaxime Ripard <maxime.ripard@free-electrons.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

Split the device_control callback of the TI OMAP DMA driver to make use
of the newly introduced callbacks, that will eventually be used to retrieve
slave capabilities.
Signed-off-by: NMaxime Ripard <maxime.ripard@free-electrons.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

The dmaengine header abbreviates destination as at least two different strings.
Make a coherent use of a single one.
Signed-off-by: NMaxime Ripard <maxime.ripard@free-electrons.com>
Acked-by: NMark Brown <broonie@kernel.org>
Acked-by: NLaurent Pinchart <laurent.pinchart@ideasonboard.com>
Acked-by: NStephen Warren <swarren@wwwdotorg.org>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

chanctnt is already filled by dma_async_device_register, which uses the channel
list to know how much channels there is.

Since it's already filled, we can safely remove it from the drivers' probe
function.
Signed-off-by: NMaxime Ripard <maxime.ripard@free-electrons.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

A platform_driver does not need to set an owner, it will be populated by the
driver core.
Signed-off-by: NWolfram Sang <wsa@the-dreams.de>

When the audio stream is paused or suspended we stop the sDMA and when it
is unpaused/resumed we start the channel without reconfiguring it.
The omap_dma_stop() clears the link configuration when we pause the dma, but
it is not setting it back on start. This will result only one audio buffer
to be played back and the DMA will stop, since the linking is disabled.
We need to restore the CLINK_CTRL register in case of resume.
Signed-off-by: NPeter Ujfalusi <peter.ujfalusi@ti.com>
Acked-by: NRussell King <rmk+kernel@arm.linux.org.uk>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

Add mb() call to resume path to ensure the necessary barrier.
Resume can happen after waking up from suspend for example.
Signed-off-by: NPeter Ujfalusi <peter.ujfalusi@ti.com>
Acked-by: NRussell King <rmk+kernel@arm.linux.org.uk>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

The argument is always set to NULL and never used. Remove it.
Signed-off-by: NLaurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

We can move the handling of the DMA synchronisation control out of the
prepare functions; this can be pre-calculated when the DMA channel has
been allocated, so we don't need to duplicate this in both prepare
functions.
Acked-by: NTony Lindgren <tony@atomide.com>
Acked-by: NVinod Koul <vinod.koul@intel.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

Move the interrupt handling for OMAP2+ into omap-dma, rather than using
the legacy support in the platform code.
Acked-by: NTony Lindgren <tony@atomide.com>
Acked-by: NVinod Koul <vinod.koul@intel.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

Export the DMA register information from the SoC specific data, such
that we can access the registers directly in omap-dma.c, mapping the
register region ourselves as well.

Rather than calculating the DMA channel register in its entirety for
each access, we pre-calculate an offset base address for the allocated
DMA channel and then just use the appropriate register offset.
Acked-by: NTony Lindgren <tony@atomide.com>
Acked-by: NVinod Koul <vinod.koul@intel.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

Provide a function to read the CSAC/CDAC register, working around the
OMAP 3.2/3.3 erratum (which requires two reads of the register if the
first returned zero.
Acked-by: NTony Lindgren <tony@atomide.com>
Acked-by: NVinod Koul <vinod.koul@intel.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

Provide a pair of channel register accessors, and a pair of global
accessors for non-channel specific registers.
Acked-by: NTony Lindgren <tony@atomide.com>
Acked-by: NVinod Koul <vinod.koul@intel.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

We don't need to read-modify-write the CCR register; we already know
what value it should contain at this point.  Use the cached CCR value
when setting the enable bit.
Acked-by: NTony Lindgren <tony@atomide.com>
Acked-by: NVinod Koul <vinod.koul@intel.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

We don't need to issue a barrier for every segment of a DMA transfer;
doing this just once per descriptor will do.
Acked-by: NTony Lindgren <tony@atomide.com>
Acked-by: NVinod Koul <vinod.koul@intel.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

Move the clnk_ctrl setup to the preparation functions, saving its
value in the omap_desc.  This only needs to be set once per descriptor,
not for each segment, so set it in omap_dma_start_desc() rather than
omap_dma_start().
Acked-by: NTony Lindgren <tony@atomide.com>
Acked-by: NVinod Koul <vinod.koul@intel.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

The only thing which changes is which registers are written, so put this
in local variables instead.  This results in smaller code.
Acked-by: NTony Lindgren <tony@atomide.com>
Acked-by: NVinod Koul <vinod.koul@intel.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

Consolidate clearing of the channel status register, rather than open
coding the same functionality in two places.
Acked-by: NTony Lindgren <tony@atomide.com>
Acked-by: NVinod Koul <vinod.koul@intel.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

Since we record the CCR register in the dma transaction, we can move the
processing of the iframe buffering errata out of the omap_dma_start().
Move it to the preparation functions.
Acked-by: NTony Lindgren <tony@atomide.com>
Acked-by: NVinod Koul <vinod.koul@intel.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

Provide our own set of more complete register definitions; this allows
us to get rid of the meaningless 1 << n constants scattered throughout
this code.
Acked-by: NTony Lindgren <tony@atomide.com>
Acked-by: NVinod Koul <vinod.koul@intel.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

Consolidate the setup of the channel control register.  Prepare the
basic value in the preparation of the DMA descriptor, and write it into
the register upon descriptor execution.
Acked-by: NTony Lindgren <tony@atomide.com>
Acked-by: NVinod Koul <vinod.koul@intel.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

Consolidate the setup of the channel source destination parameters
register.  This way, we calculate the required CSDP value when we setup
a transfer descriptor, and only write it to the device registers once
when we start the descriptor.
Acked-by: NTony Lindgren <tony@atomide.com>
Acked-by: NVinod Koul <vinod.koul@intel.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

Read the current DMA position from the hardware directly rather than via
arch/arm/plat-omap/dma.c.
Acked-by: NTony Lindgren <tony@atomide.com>
Acked-by: NVinod Koul <vinod.koul@intel.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

Program the non-cyclic mode DMA start/stop directly, rather than via
arch/arm/plat-omap/dma.c.
Acked-by: NTony Lindgren <tony@atomide.com>
Acked-by: NVinod Koul <vinod.koul@intel.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>