Currently drivers are limited to 19 slots for cyclic transfers.
However, if the DMA burst size is the same as the period size,
the period size can be changed to the full buffer size and
intermediate interrupts activated. Since intermediate interrupts
will trigger for each burst and the burst size is the same as
the period size, the driver will get interrupts each period as
expected. This has the benefit of allowing the functionality of
many more slots, but only uses 2 slots.

This workaround is only active if more than 19 slots are needed
and the burst size matches the period size.
Acked-by: NPeter Ujfalusi <peter.ujfalusi@ti.com>
Signed-off-by: NJohn Ogness <john.ogness@linutronix.de>
Signed-off-by: NSekhar Nori <nsekhar@ti.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

The dynamic or on demand pm_runtime does not work correctly on am335x and
am437x due to interference with hwmod.
Fall back using the pm_runtime usage as it was in the old driver stack,
meaning that at probe time call pm_runtime_enable() and
pm_runtime_get_sync() for the TPTCs as well.

Fixes: 1be5336b ("dmaengine: edma: New device tree binding")
Signed-off-by: NPeter Ujfalusi <peter.ujfalusi@ti.com>
Reported-by: NTero Kristo <t-kristo@ti.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

In order to avoid possible race condition when client drivers are using
dmaengine_terminate_sync() call to disable the channel.
Signed-off-by: NPeter Ujfalusi <peter.ujfalusi@ti.com>
Suggested-by: NLars-Peter Clausen <lars@metafoo.de>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

We need the callback to support the dmaengine_terminate_sync().
Signed-off-by: NPeter Ujfalusi <peter.ujfalusi@ti.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

When retrieving the residue value, the SRC/DST fields of the
active PaRAM are read to determine the current position of
the DMA engine. However, the AM335x Technical Reference Manual
states:

  11.3.3.6 Parameter Set Updates

  After the TR is read from the PaRAM (and is in the process
  of being submitted to the EDMA3TC), the following fields are
  updated as needed: ... SRC DST

This means SRC/DST is incremented even though the DMA transfer
may not have started yet or is in progress. Thus if the reader
of the residue accesses the DMA buffer too quickly, the CPU is
misinformed about the data that has been successfully processed.

The CCSTAT.ACTV register is a boolean that is set if any TR is
being processed by either the EMDA3CC or EDMA3TC. By polling
this register it is possible to ensure that the residue value
returned is valid for immediate processing. However, since the
DMA engine may be active, polling may never hit a moment where
no TR is being processed. To handle this, the SRC/DST is also
polled to see if it changes. And as a last resort, a max loop
count for the busy waiting exists to avoid an infinite loop.
Signed-off-by: NJohn Ogness <john.ogness@linutronix.de>
Acked-by: NPeter Ujfalusi <peter.ujfalusi@ti.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

edma_alloc_slot was not checking the channel mapping support existence when
slot 0 has been requested (used as entry slot for channel/event 0).
Signed-off-by: NPeter Ujfalusi <peter.ujfalusi@ti.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

Add support for providing device to filter_fn mapping so client drivers
can switch to use the dma_request_chan() API.
Signed-off-by: NPeter Ujfalusi <peter.ujfalusi@ti.com>
Reviewed-by: NArnd Bergmann <arnd@arndb.de>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

Due to changes in device and platform code drivers w/o probe will fail to
load. This means that the devices for eDMA TPTCs are goign to be without
driver and omap hwmod code will turn them off after the kernel finished
loading:
[    3.015900] platform 49800000.tptc: omap_device_late_idle: enabled but no driver.  Idling
[    3.024671] platform 49a00000.tptc: omap_device_late_idle: enabled but no driver.  Idling

This will prevent eDMA to work since the TPTCs are not enabled.
Signed-off-by: NPeter Ujfalusi <peter.ujfalusi@ti.com>
Fixes: 34635b1a ("dmaengine: edma: Add dummy driver skeleton for edma3-tptc")
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

This change makes the DT file to be easier to read since the reserved slots
array does not need the '/bits/ 16' to be specified, which might confuse
some people.
Signed-off-by: NPeter Ujfalusi <peter.ujfalusi@ti.com>
Acked-by: NArnd Bergmann <arnd@arndb.de>
Acked-by: NRob Herring <robh@kernel.org>
Acked-by: NTony Lindgren <tony@atomide.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

This change makes the DT file to be easier to read since the memcpy
channels array does not need the '/bits/ 16' to be specified, which might
confuse some people.
Signed-off-by: NPeter Ujfalusi <peter.ujfalusi@ti.com>
Acked-by: NArnd Bergmann <arnd@arndb.de>
Acked-by: NRob Herring <robh@kernel.org>
Acked-by: NTony Lindgren <tony@atomide.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

The current code uses bits 0-2 instead of 4-6 as the comment says.

Fixes: 633e42b8 ('dmaengine: edma: Get qDMA channel information from HW also')
Signed-off-by: NDan Carpenter <dan.carpenter@oracle.com>
Acked-by: NPeter Ujfalusi <peter.ujfalusi@ti.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

During the edma rework, a build error was introduced for the
case that CONFIG_OF is disabled:

drivers/built-in.o: In function `edma_tc_set_pm_state':
:(.text+0x43bf0): undefined reference to `of_find_device_by_node'

As the edma_tc_set_pm_state() function does nothing in case
we are running without OF, this adds an IS_ENABLED() check
that turns the function into an empty stub then and avoids the
link error.
Signed-off-by: NArnd Bergmann <arnd@arndb.de>
Fixes: ca304fa9 ("ARM/dmaengine: edma: Public API to use private struct pointer")
Acked-by: NPeter Ujfalusi <peter.ujfalusi@ti.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

The eDMA3 TPTC does not need any software configuration, but it is a
separate IP block in the SoC. In order the omap hwmod core to be able to
handle the TPTC resources correctly in regards of PM we need to have a
driver loaded for it.
This patch will add a dummy driver skeleton without probe or remove
callbacks provided.
Signed-off-by: NPeter Ujfalusi <peter.ujfalusi@ti.com>
Reported-by: NOlof Johansson <olof@lixom.net>
Tested-by: NFelipe Balbi <balbi@ti.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

With the old binding and driver architecture we had many issues:
No way to assign eDMA channels to event queues, thus not able to tune the
system by moving specific DMA channels to low/high priority servicing. We
moved the cyclic channels to high priority within the code, but that was
just a workaround to this issue.
Memcopy was fundamentally broken: even if the driver scanned the DT/devices
in the booted system for direct DMA users (which is not effective when the
events are going through a crossbar) and created a map of 'used' channels,
this information was not really usable. Since via dmaengien API the eDMA
driver will be called with _some_ channel number, we would try to request
this channel when any channel is requested for memcpy. By luck we got
channel which is not used by any device most of the time so things worked,
but if a device would have been using the given channel, but not requested
it, the memcpy channel would have been waiting for HW event.
The old code had the am33xx/am43xx DMA event router handling embedded. This
should have been done in a separate driver since it is not part of the
actual eDMA IP.
There were no way to 'lock' PaRAM slots to be used by the DSP for example
when booting with DT.
In DT boot the edma node used more than one hwmod which is not a good
practice and the kernel prints warning because of this.

With the new bindings and the changes in the driver we can:
- No regression with Legacy binding and non DT boot
- DMA channels can be assigned to any TC (to set priority)
- PaRAM slots can be reserved for other cores to use
- Dynamic power management for CC and TCs, if only TC0 is used all other TC
  can be powered down for example
Signed-off-by: NPeter Ujfalusi <peter.ujfalusi@ti.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

Instead of nesting functions just merge them since the resulting function
is still small and readable.
Signed-off-by: NPeter Ujfalusi <peter.ujfalusi@ti.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

The channel/slot reservation is not supported when booted with DT so there
is not need to allocate memory.
Signed-off-by: NPeter Ujfalusi <peter.ujfalusi@ti.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

Move all code under one function to do the dma device and eDMA channel
related setup so they are not scattered around the driver.
Signed-off-by: NPeter Ujfalusi <peter.ujfalusi@ti.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

Query the number of qDMA channels from CCCFG register.
Signed-off-by: NPeter Ujfalusi <peter.ujfalusi@ti.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

edma_assign_channel_eventq() is a wrapper around edma_map_dmach_to_queue()
We can merge the content of the later so we will have only one function
to be used for mapping channels to given eventq
Signed-off-by: NPeter Ujfalusi <peter.ujfalusi@ti.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

These inline functions are designed to modify parts of the PaRAM in eDMA.
Change the names accordingly.
Signed-off-by: NPeter Ujfalusi <peter.ujfalusi@ti.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

Instead of passing a pointer to struct edma_cc and the channel number,
pass only the pointer to the edma_chan structure for the given channel.
This struct contains all the information needed by the functions and the
use of this makes it obvious that most of the sanity checks can be removed
from the driver.
Signed-off-by: NPeter Ujfalusi <peter.ujfalusi@ti.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

If the transfer is shorted then 64K we can complete it with one ACNT burst
by configuring ACNT to the length of the copy, this require one paRAM slot.
Otherwise we use two paRAM slots for the copy:
slot1: will copy (length / 32767) number of 32767 byte long blocks
slot2: will be configured to copy the remaining data.

According to tests this patch increases the throughput of memcpy from
~3MB/s to 15MB/s
Signed-off-by: NPeter Ujfalusi <peter.ujfalusi@ti.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

Despite the claim by the original commit adding the memcpy
support, eDMA does not have constraint on the alignment of src, dst
or length in increment mode.
Signed-off-by: NPeter Ujfalusi <peter.ujfalusi@ti.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

If the eDMA3 has support for channel paRAM slot mapping we can utilize it
to allocate slots on demand and save precious slots for real transfers.
On am335x the eDMA has 64 channels which means we can unlock 64 paRAM
slots out from the available 256.
Signed-off-by: NPeter Ujfalusi <peter.ujfalusi@ti.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

The names chosen for the bitfields were quite confusing and given no real
information on what they are used for...

edma_inuse -> slot_inuse: tracks the slot usage/availability
edma_unused -> channel_unused: tracks the channel usage/availability
Signed-off-by: NPeter Ujfalusi <peter.ujfalusi@ti.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

Instead of directly reading it from CCCFG register take the information out
once when we set up the configuration from the HW.
Signed-off-by: NPeter Ujfalusi <peter.ujfalusi@ti.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

No need to run through the bits in QEMR and CCERR events since they will
not trigger any action, so just clearing the errors there is fine.
In case of the missed event the loop can be optimized so we spend less time
to handle the event.
Signed-off-by: NPeter Ujfalusi <peter.ujfalusi@ti.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

In the ccerr interrupt handler the code checks for pending errors in the
error status registers in two different places.
Move the check out to a helper function.
Signed-off-by: NPeter Ujfalusi <peter.ujfalusi@ti.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

With the merger of the arch/arm/common/edma.c code into the dmaengine
driver, there is no longer need to have per channel callback/data storage
for interrupt events.
Signed-off-by: NPeter Ujfalusi <peter.ujfalusi@ti.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

Remove or rewrite the comments for the internal functions.
Signed-off-by: NPeter Ujfalusi <peter.ujfalusi@ti.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

Warning message in case of linking between paRAM slots in different eDMA
controllers.
Signed-off-by: NPeter Ujfalusi <peter.ujfalusi@ti.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

edma_write_slot() is for writing an entire paRAM slot.
Signed-off-by: NPeter Ujfalusi <peter.ujfalusi@ti.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

We have access to dev, so it is better to use the dev_dbg for debug prints.
Signed-off-by: NPeter Ujfalusi <peter.ujfalusi@ti.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

Be consistent and do not mix the use of dev, &pdev->dev, etc in the
functions.
Signed-off-by: NPeter Ujfalusi <peter.ujfalusi@ti.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

When allocating a memory for number of items it is better (looks better)
to use devm_kcalloc.
Signed-off-by: NPeter Ujfalusi <peter.ujfalusi@ti.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

Instead of using defines to specify the size of different arrays and
bitmaps, allocate the memory for them based on the information we get from
the HW itself.
Since these defines are set based on the worst case, there are devices
where they are not valid.
Signed-off-by: NPeter Ujfalusi <peter.ujfalusi@ti.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

Move the code out from arch/arm/common and merge it inside of the dmaengine
driver.
This change is done with as minimal (if eny) functional change to the code
as possible to avoid introducing regression.
Signed-off-by: NPeter Ujfalusi <peter.ujfalusi@ti.com>
Acked-by: NTony Lindgren <tony@atomide.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

Since the driver stack no longer depends on lookup with id number in a
global array of pointers, the limitation for the number of eDMAs are no
longer needed. We can handle as many eDMAs in legacy and DT boot as we have
memory for them to allocate the needed structures.
Signed-off-by: NPeter Ujfalusi <peter.ujfalusi@ti.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

Instead of relying on indexes pointing to edma private date in the global
pointer array, pass the private data pointer via the public API.
Signed-off-by: NPeter Ujfalusi <peter.ujfalusi@ti.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

If the of_dma_controller is registered in the non dmaengine driver we could
have race condition:
the of_dma_controller has been registered, but the dmaengine driver is not
yet probed. Drivers requesting DMA channels during this window will fail
since we do not yet have dmaengine drivers registered.
Signed-off-by: NPeter Ujfalusi <peter.ujfalusi@ti.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>