dmaengine routines can be called from interrupt context and with interrupts
disabled.  Whereas spin_unlock_bh can't be called from such contexts. So this
patch converts all spin_*_bh routines to irqsave variants.

Also, spin_lock() used in tasklet is converted to irqsave variants, as tasklet
can be interrupted, and dma requests from such interruptions may also call
spin_lock.

Now, submission from callbacks are permitted as per dmaengine framework. So we
shouldn't hold any locks while calling callbacks. As locks were taken by parent
routines, so releasing them before calling callbacks doesn't look clean enough.
So, locks are taken inside all routine now, whereever they are required. And
dwc_descriptor_complete is always called without taking locks.
Signed-off-by: NViresh Kumar <viresh.kumar@st.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

If len passed in sg for slave_sg transfers is greater than DWC_MAX_COUNT, then
driver programmes controller incorrectly.  This patch adds code to handle this
situation by allocation more than one desc for same sg.
Signed-off-by: NViresh Kumar <viresh.kumar@st.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

If transfer status is !=DMA_SUCCESS, return total transfer len as residue,
instead of zero.
Signed-off-by: NViresh Kumar <viresh.kumar@st.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

If dmaengine_terminate_all() is called for dma channel, then it doesn't make
much sense to call registered callback routine. While in case of success or
failure it must be called.
Signed-off-by: NViresh Kumar <viresh.kumar@st.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

With the addition of the "pause" feature, an active wait was introduced
to check the "FIFO empty" event. This event was not always happening and
a timout contition was needed.
But, in some cases, this event depend on the peripheral connected to the
channel that is paused: FIFO becomes empty if the peripheral consumes data.
The timeout is pretty difficult to evaluate. Moreover, this check is not
needed.
In conclusion, it seems sensible to entirely remove the checking of
"FIFO empty" status when pausing.
Signed-off-by: NNicolas Ferre <nicolas.ferre@atmel.com>
[commit msg edited for grammer]
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

Signed-off-by: NTomoya MORINAGA <tomoya-linux@dsn.okisemi.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

Support new device OKI SEMICONDUCTOR ML7223 IOH(Input/Output Hub).
The ML7223 IOH is for MP(Media Phone) use.
The ML7223 is companion chip for Intel Atom E6xx series.
The ML7223 is completely compatible for Intel EG20T PCH.
Signed-off-by: NTomoya MORINAGA <tomoya-linux@dsn.okisemi.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

Support I2S device for ML7213 IOH
Signed-off-by: NTomoya MORINAGA <tomoya-linux@dsn.okisemi.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

Currently, Direct-Start mode(*) is enabled.
Our IOH's devices must not use this mode.
This causes unexpected behavior.
This patch deletes Direct-Start setting.
(*) This mode is used in order for CPU to generate the DMA request.
Signed-off-by: NTomoya MORINAGA <tomoya-linux@dsn.okisemi.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

Fix checkpatch warnings.
Signed-off-by: NTomoya MORINAGA <tomoya-linux@dsn.okisemi.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

Currently, even-channel number is set as tx direction and odd is set as rx.
However, though video-in uses ch6, the direction is not tx but rx.
This patch sets video-in's DMA direction correctly.
Signed-off-by: NTomoya MORINAGA <tomoya-linux@dsn.okisemi.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

A little function helps to chain descriptors:
it is already used in cyclic dma operations, now use it in memcpy and slave_sg
preparation functions.
Signed-off-by: NNicolas Ferre <nicolas.ferre@atmel.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

Pause and resume controls are useful for audio devices. This also returns
correct status from atc_tx_status() in case chan is paused.

Idea from dw_dmac patch by Linus Walleij.
Signed-off-by: NNicolas Ferre <nicolas.ferre@atmel.com>
Acked-by: NLinus Walleij <linus.walleij@linaro.org>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

If transfer status is !=DMA_SUCCESS, return total transfer len as residue,
instead of zero.

Idea from dw_dmac patch by Viresh Kumar.
Signed-off-by: NNicolas Ferre <nicolas.ferre@atmel.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

The compiler nowadays moans about possibly non-assigned variable.
Fix this by default-assigning 0.
Signed-off-by: NPhilippe Langlais <philippe.langlais@linaro.org>
Signed-off-by: NLinus Walleij <linus.walleij@linaro.org>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

DMA controller has two AHB interfaces on the SOC internal
matrix.
It is more efficient to specialize each interface as the
access to memory can introduce latencies that are not compatible
with peripheral accesses requirements.
Signed-off-by: NNicolas Ferre <nicolas.ferre@atmel.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

There is no need to test if channel is enabled in tasklet:
- in error path, channel is disabled in interrupt routine
- in normal path, this test is performed in sub functions to report
a misuse of the engine.
Signed-off-by: NNicolas Ferre <nicolas.ferre@atmel.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

Signed-off-by: NNicolas Ferre <nicolas.ferre@atmel.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

Signed-off-by: NNicolas Ferre <nicolas.ferre@atmel.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

Now we use Buffer Transfer Completed interrupts. If we
want a chained buffer completed information, we setup the
ATC_IEN bit in CTRLB register in the lli.
This is done by set_desc_eol() function and used by
memcpy/slave_sg functions.
Signed-off-by: NNicolas Ferre <nicolas.ferre@atmel.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

In the function pdc_desc_get(), var 'i' is not
initialized before use. This patch fixes it.
Signed-off-by: NLiu Yuan <tailai.ly@taobao.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

Use the correct api in probe to enable runtime pm for this driver.
Additionally, do not just call legacy suspend for runtime_suspend,
as this duplicates some work the pci core does for you.
Signed-off-by: NKristen Carlson Accardi <kristen@linux.intel.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

This patch makes BUG_ON() usage correct in drivers/dma/ppc4xx/adam.c

Cc: Vinod Koul <vinod.koul@intel.com>
Cc: Dan Williams <dan.j.williams@intel.com>
Cc: Grant Likely <grant.likely@secretlab.ca>
Cc: Anatolij Gustschin <agust@denx.de>
Cc: Sean MacLennan <smaclennan@pikatech.com>
Cc: Joe Perches <joe@perches.com>
Signed-off-by: NColy Li <bosong.ly@taobao.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

This patch makes BUG_ON() usage correct in drivers/dma/mv_xor.c

Cc: Vinod Koul <vinod.koul@intel.com>
Cc: Dan Williams <dan.j.williams@intel.com>
Signed-off-by: NColy Li <bosong.ly@taobao.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

This patch makes BUG_ON() usage correct in drivers/dma/iop-adma.c.

Cc: Dan Williams <dan.j.williams@intel.com>
Cc: Vinod Koul <vinod.koul@intel.com>
Signed-off-by: NColy Li <bosong.ly@taobao.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

dwc_scan_descriptors scans all descriptors from active_list in case transfer is
not completed.  It compares first_desc->lli.llp, and then all childrens of its
tx_list. But it doesn't compare its own address, i.e. first_desc->txd.phys, as
this is what we have initially programmed into the controller register. So this
causes dma to stop and finish a transfer, which was never started. And thus
fail.
Signed-off-by: NViresh Kumar <viresh.kumar@st.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

Convert to the new irq chip functions and cleanup the name space.

[ Guennadi reported: irq_data_get_chip_data is undefined. Yes, I screwed up. 
 it needs to be irq_data_get_irq_chip_data ]
Signed-off-by: NThomas Gleixner <tglx@linutronix.de>
Tested-by: NGuennadi Liakhovetski <g.liakhovetski@gmx.de>
Cc: Dan Williams <dan.j.williams@intel.com>
LKML-Reference: <alpine.LFD.2.00.1103251220000.31464@localhost6.localdomain6>

The cell's platform_data is now accessed with a helper function;
change clients to use that, and remove the now-unused data_size.

Note that the mfd's platform_data is marked __devinitdata.  This
is still correct in all cases except for the timbgpio driver, whose
remove hook has been changed to no longer reference the pdata.
Signed-off-by: NAndres Salomon <dilinger@queued.net>
Signed-off-by: NSamuel Ortiz <sameo@linux.intel.com>

When we try to test all channels present on our controller together, some
channels of lower priority may be very slow as compared to others. If number of
transfers is unlimited, some channels may timeout and will not finish within 3
seconds. Thus, while doing such regress testing we may need to have higher value
of timeouts. This patch adds support for passing timeout value via module
parameters. Default value is 3 msec, a negative value means max timeout
possible.
Signed-off-by: NViresh Kumar <viresh.kumar@st.com>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

As a side effect this makes IMX_DMA selectable on i.MX21 again, because
the symbol ARCH_MX21 doesn't exist (MACH_MX21 would have been more correct).
Signed-off-by: NUwe Kleine-König <u.kleine-koenig@pengutronix.de>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

Replace EXTRA_CFLAGS with ccflags-y.
Signed-off-by: Nmatt mooney <mfm@muteddisk.com>
Acked-by: NWANG Cong <xiyou.wangcong@gmail.com>
Signed-off-by: NMichal Marek <mmarek@suse.cz>

The original dma_halt() function set the CA (channel abort) bit on both
the 83xx and 85xx controllers. This is incorrect on the 83xx, where this
bit means TEM (transfer error mask) instead. The 83xx doesn't support
channel abort, so we only do this operation on 85xx.
Signed-off-by: NIra W. Snyder <iws@ovro.caltech.edu>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

This merges the fsl_chan_ld_cleanup() function into the dma_do_tasklet()
function to reduce locking overhead. In the best case, we will be able
to keep the DMA controller busy while we are freeing used descriptors.
In all cases, the spinlock is grabbed two times fewer than before on
each transaction.
Signed-off-by: NIra W. Snyder <iws@ovro.caltech.edu>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

Previous to this patch, the dma_run_dependencies() function has been
called while holding desc_lock. This function can call tx_submit() for
other descriptors, which may try to re-grab the lock. Avoid this by
moving the descriptors to be cleaned up to a temporary list, and
dropping the lock before cleanup.

At the same time, add support for automatic unmapping of src and dst
buffers, as offered by the DMAEngine API.
Signed-off-by: NIra W. Snyder <iws@ovro.caltech.edu>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

Enabling poisoning in the dmapool API quickly showed that the DMA
controller was fetching descriptors that should not have been in use.
This has caused intermittent controller lockups during testing.

I have been unable to figure out the exact set of conditions which cause
this to happen. However, I believe it is related to the driver using the
hardware registers to track whether the controller is busy or not. The
code can incorrectly decide that the hardware is idle due to lag between
register writes and the hardware actually becoming busy.

To fix this, the driver has been reworked to explicitly track the state
of the hardware, rather than try to guess what it is doing based on the
register values.

This has passed dmatest with 10 threads per channel, 100000 iterations
per thread several times without error. Previously, this would fail
within a few seconds.
Signed-off-by: NIra W. Snyder <iws@ovro.caltech.edu>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

This fixes some minor violations of the coding style. It also changes
the style of the device_prep_dma_*() function definitions so they are
identical.
Signed-off-by: NIra W. Snyder <iws@ovro.caltech.edu>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

This adds better tracking to link descriptor allocations, callbacks, and
frees. This makes it much easier to track errors with link descriptors.
Signed-off-by: NIra W. Snyder <iws@ovro.caltech.edu>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

This makes debugging the driver much easier when multiple channels are
running concurrently. In addition, you can see how much descriptor
memory each channel has allocated via the dmapool API in sysfs.
Signed-off-by: NIra W. Snyder <iws@ovro.caltech.edu>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

This is a purely cosmetic cleanup. It is nice to have related functions
right next to each other in the code.
Signed-off-by: NIra W. Snyder <iws@ovro.caltech.edu>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

The dmatest code relies on the DMAEngine API to automatically call
dma_unmap_single() on src buffers. The flags it passes are incorrect,
fix them.
Signed-off-by: NIra W. Snyder <iws@ovro.caltech.edu>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>