The 83xx controller has external start capability, but lacks external pause
capability. Hook up the external start function pointer for the 83xx
controller.
Signed-off-by: NIra W. Snyder <iws@ovro.caltech.edu>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

By default, the Freescale 83xx DMA controller uses the PCI Read Line
command when reading data over the PCI bus. Setting the controller to use
the PCI Read Multiple command instead allows the controller to read much
larger bursts of data, which provides a drastic speed increase.

The slowdown due to using PCI Read Line was only observed when a PCI-to-PCI
bridge was between the devices trying to communicate.

A simple test driver showed an increase from 4MB/sec to 116MB/sec when
performing DMA over the PCI bus. Using DMA to transfer between blocks of
local SDRAM showed no change in performance with this patch. The dmatest
driver was also used to verify the correctness of the transfers, and showed
no errors.
Signed-off-by: NIra W. Snyder <iws@ovro.caltech.edu>
Acked-by: NTimur Tabi <timur@freescale.com>
Acked-by: NKumar Gala <galak@kernel.crashing.org>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

We we build with dma_addr_t as a 64-bit quantity we get:

drivers/dma/fsldma.c: In function 'fsl_chan_xfer_ld_queue':
drivers/dma/fsldma.c:625: warning: cast to pointer from integer of different size
drivers/dma/fsldma.c: In function 'fsl_dma_chan_do_interrupt':
drivers/dma/fsldma.c:737: warning: cast to pointer from integer of different size
drivers/dma/fsldma.c:737: warning: cast to pointer from integer of different size
drivers/dma/fsldma.c: In function 'of_fsl_dma_probe':
drivers/dma/fsldma.c:927: warning: cast to pointer from integer of different
Signed-off-by: NKumar Gala <galak@kernel.crashing.org>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

When preparing a memcpy operation, if the kernel fails to allocate memory
for a link descriptor after the first link descriptor has already been
allocated, then some memory will never be released. Fix the problem by
walking the list of allocated descriptors backwards, and freeing the
allocated descriptors back into the DMA pool.
Signed-off-by: NIra W. Snyder <iws@ovro.caltech.edu>
Signed-off-by: NLi Yang <leoli@freescale.com>

On the 83xx controller, snooping is necessary for the DMA controller to
ensure cache coherence with the CPU when transferring to/from RAM.

The last descriptor in a chain will always have the End-of-Chain interrupt
bit set, so we can set the snoop bit while adding the End-of-Chain
interrupt bit.
Signed-off-by: NIra W. Snyder <iws@ovro.caltech.edu>
Signed-off-by: NLi Yang <leoli@freescale.com>

When creating a DMA transaction with multiple descriptors, the async_tx
cookie is set to 0 for each descriptor in the chain, excluding the last
descriptor, whose cookie is set to -EBUSY.

When fsl_dma_tx_submit() is run, it only assigns a cookie to the first
descriptor. All of the remaining descriptors keep their original value,
including the last descriptor, which is set to -EBUSY.

After the DMA completes, the driver will update the last completed cookie
to be -EBUSY, which is an error code instead of a valid cookie. This causes
dma_async_is_complete() to always return DMA_IN_PROGRESS.

This causes the fsldma driver to never cleanup the queue of link
descriptors, and the driver will re-run the DMA transaction on the hardware
each time it receives the End-of-Chain interrupt. This causes an infinite
loop.

With this patch, fsl_dma_tx_submit() is changed to assign a cookie to every
descriptor in the chain. The rest of the code then works without problems.
Signed-off-by: NIra W. Snyder <iws@ovro.caltech.edu>
Signed-off-by: NLi Yang <leoli@freescale.com>

When using the DMA controller from multiple threads at the same time, it is
possible to get lots of "DMA halt timeout!" errors printed to the kernel
log.

This occurs due to a race between fsl_dma_memcpy_issue_pending() and the
interrupt handler, fsl_dma_chan_do_interrupt(). Both call the
fsl_chan_xfer_ld_queue() function, which does not protect against
concurrent accesses to dma_halt() and dma_start().

The existing spinlock is moved to cover the dma_halt() and dma_start()
functions. Testing shows that the "DMA halt timeout!" errors disappear.
Signed-off-by: NIra W. Snyder <iws@ovro.caltech.edu>
Signed-off-by: NLi Yang <leoli@freescale.com>

Fix the check of potential array overflow when using corrupted channel
device tree nodes.
Signed-off-by: NRoel Kluin <roel.kluin@gmail.com>
Signed-off-by: NLi Yang <leoli@freescale.com>

Centralize this common initialization (and one case where ipu_idmac is
duplicating ->chan initialization).
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

Prevent dev_err from firing even if we successfully detected 'dma-idle'
before the full 1ms timeout has elapsed.
Acked-by: NRoel Kluin <roel.kluin@gmail.com>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

The mpc83xx variant uses a shared IRQ for all channels, so the individual
channel nodes don't have an interrupt property. Fix the code to print the
controller IRQ instead if there isn't any for the channel.
Acked-by: NTimur Tabi <timur@freescale.com>
Acked-by: NLi Yang <leoli@freescale.com>
Signed-off-by: NPeter Korsgaard <jacmet@sunsite.dk>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

There's no per-channel IRQ on mpc83xx, so only call free_irq if we have one.
Acked-by: NTimur Tabi <timur@freescale.com>
Acked-by: NLi Yang <leoli@freescale.com>
Signed-off-by: NPeter Korsgaard <jacmet@sunsite.dk>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

The dmaengine sysfs implementation was fixed to support proper
lifetime rules which means that the current:

new_fsl_chan->dev = &new_fsl_chan->common.dev->device;

...retrieves a NULL pointer because new_fsl_chan->common.dev has not
been allocated at this point.  So, set new_fsl_chan->dev to a valid
device.

Cc: Li Yang <leoli@freescale.com>
Cc: Zhang Wei <zw@zh-kernel.org>
Reported-by: NIra Snyder <iws@ovro.caltech.edu>
Tested-by: NIra Snyder <iws@ovro.caltech.edu>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

Resolves:
WARNING: at drivers/base/core.c:122 device_release+0x4d/0x52()
Device 'dma0chan0' does not have a release() function, it is broken and must be fixed.

The dma_chan_dev object is introduced to gear-match sysfs kobject and
dmaengine channel lifetimes.  When a channel is removed access to the
sysfs entries return -ENODEV until the kobject can be released.

The bulk of the change is updates to existing code to handle the extra
layer of indirection between a dma_chan and its struct device.
Reported-by: NAlexander Beregalov <a.beregalov@gmail.com>
Acked-by: NStephen Hemminger <shemminger@vyatta.com>
Cc: Haavard Skinnemoen <haavard.skinnemoen@atmel.com>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

All users have been converted to either the general-purpose allocator,
dma_find_channel, or dma_request_channel.
Reviewed-by: NAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

Modify the Freescale Elo / Elo Plus DMA driver so that it can be compiled as
a module.

The primary change is to stop treating the DMA controller as a bus, and the
DMA channels as devices on the bus.  This is because the Open Firmware (OF)
kernel code does not allow busses to be removed, so although we can call
of_platform_bus_probe() to probe the DMA channels, there is no
of_platform_bus_remove().  Instead, the DMA channels are manually probed,
similar to what fsl_elbc_nand.c does.

Cc: Scott Wood <scottwood@freescale.com>
Acked-by: NLi Yang <leoli@freescale.com>
Signed-off-by: NTimur Tabi <timur@freescale.com>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

The Freescale Elo DMA driver runs an internal self-test before registering
the channels with the DMA engine.  This self-test has a fundemental flaw in
that it calls the DMA engine's callback functions directly before the
registration.  However, the registration initializes some variables that the
callback functions uses, namely the device struct.

The code works today because there are two device structs: the one created
by the DMA engine, and one created by the Open Firmware (OF) subsystem.  The
self-test currently uses the device struct created by OF.  However, in the
future, some of the device structs created by OF will be eliminated.
This means that the self-test will only have access to the device struct
created by the DMA engine.  But this device struct isn't initialized when
the self-test runs, and this causes a kernel panic.

Since there is already a DMA test module (dmatest), the internal self-test
code is not useful anyway.  It is extremely unlikely that the test will fail
in normal usage.  It may have been helpful during development, but not any more.

Cc: Kumar Gala <galak@kernel.crashing.org>
Cc: Li Yang <leoli@freescale.com>
Cc: Scott Wood <scottwood@freescale.com>
Signed-off-by: NTimur Tabi <timur@freescale.com>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

A DMA controller capable of doing slave transfers may need to know a
few things about the slave when preparing the channel. We don't want
to add this information to struct dma_channel since the channel hasn't
yet been bound to a client at this point.

Instead, pass a reference to the client requesting the channel to the
driver's device_alloc_chan_resources hook so that it can pick the
necessary information from the dma_client struct by itself.

[dan.j.williams@intel.com: fixed up fsldma and mv_xor]
Acked-by: NMaciej Sosnowski <maciej.sosnowski@intel.com>
Signed-off-by: NHaavard Skinnemoen <haavard.skinnemoen@atmel.com>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

Signed-off-by: NLi Yang <leoli@freescale.com>
Acked-by: NZhang Wei <zw@zh-kernel.org>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

'ack' is currently a simple integer that flags whether or not a client is done
touching fields in the given descriptor.  It is effectively just a single bit
of information.  Converting this to a flags parameter allows the other bits to
be put to use to control completion actions, like dma-unmap, and capture
results, like xor-zero-sum == 0.

Changes are one of:
1/ convert all open-coded ->ack manipulations to use async_tx_ack
   and async_tx_test_ack.
2/ set the ack bit at prep time where possible
3/ make drivers store the flags at prep time
4/ add flags to the device_prep_dma_interrupt prototype
Acked-by: NMaciej Sosnowski <maciej.sosnowski@intel.com>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

DMA drivers no longer need to be notified of dependency submission
events as async_tx_run_dependencies and async_tx_channel_switch will
handle the scheduling and execution of dependent operations.

[sfr@canb.auug.org.au: extend this for fsldma]
Acked-by: NShannon Nelson <shannon.nelson@intel.com>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

Split MPC83xx EOCDI event from MPC85xx EOLNI event, which is
also need to update cookie and start the next transfer.
The DMA channel irq handler function code is refined.
The patch is tested on MPC8377MDS board.
Signed-off-by: NZhang Wei <wei.zhang@freescale.com>
Signed-off-by; Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

Always enabling the fsl_dma_self_test() to ensure the DMA controller
should works well after the driver probed.
Signed-off-by: NZhang Wei <wei.zhang@freescale.com>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

Documentation/powerpc/booting-without-of.txt specifies the
compatiables we should bind to for this driver (elo, eloplus).
Use these instead of the extremely specific 'mpc8540' and 'mpc8349'
compatiables.
Acked-by: NDan Williams <dan.j.williams@intel.com>
Signed-off-by: NKumar Gala <galak@kernel.crashing.org>

The DMA_INTERRUPT async_tx is a NULL transfer, thus the BCR(count register)
is 0. When the transfer started with a byte count of zero, the DMA
controller will triger a PE(programming error) event and halt, not a normal
interrupt. I add special codes for PE event and DMA_INTERRUPT
async_tx testing.
Signed-off-by: NZhang Wei <wei.zhang@freescale.com>
Signed-off-by: NAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

The patch 'fsldma: do not cleanup descriptors in hardirq context'
(commit 222ccf9a) removed descriptors
cleanup function to tasklet but the completed cookie do not updated.
Thus, the DMA controller will get lots of duplicated transfer
interrupts. Just make a completed cookie update in interrupt handler.
And keep other cleanup jobs in tasklet function.
Tested-by: NSebastian Siewior <bigeasy@linutronix.de>
Signed-off-by: NZhang Wei <wei.zhang@freescale.com>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

This is a bug that I assigned DMA_INTERRUPT capability to fsldma
but missing device_prep_dma_interrupt function. For a bug in
dmaengine.c the driver passed BUG_ON() checking. The patch fixes it.
Signed-off-by: NZhang Wei <wei.zhang@freescale.com>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

There are warning messages reported by Stephen Rothwell with
ARCH=powerpc allmodconfig build:

drivers/dma/fsldma.c: In function 'fsl_dma_prep_memcpy':
drivers/dma/fsldma.c:439: warning: comparison of distinct pointer types
lacks a cast
drivers/dma/fsldma.c: In function 'fsl_chan_xfer_ld_queue':
drivers/dma/fsldma.c:584: warning: format '%016llx' expects type 'long long
unsigned int', but argument 4 has type 'dma_addr_t'
drivers/dma/fsldma.c: In function 'fsl_dma_chan_do_interrupt':
drivers/dma/fsldma.c:668: warning: format '%x' expects type 'unsigned int',
but argument 5 has type 'dma_addr_t'
drivers/dma/fsldma.c:684: warning: format '%016llx' expects type 'long long
unsigned int', but argument 4 has type 'dma_addr_t'
drivers/dma/fsldma.c:684: warning: format '%016llx' expects type 'long long
unsigned int', but argument 5 has type 'dma_addr_t'
drivers/dma/fsldma.c:701: warning: format '%02x' expects type 'unsigned
int', but argument 4 has type 'dma_addr_t'
drivers/dma/fsldma.c: In function 'fsl_dma_self_test':
drivers/dma/fsldma.c:840: warning: format '%d' expects type 'int', but
argument 5 has type 'size_t'
drivers/dma/fsldma.c: In function 'of_fsl_dma_probe':
drivers/dma/fsldma.c:1010: warning: format '%08x' expects type 'unsigned
int', but argument 5 has type 'resource_size_t'

This patch fixed the above warning messages.
Signed-off-by: NZhang Wei <wei.zhang@freescale.com>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

"Cleaning" descriptors involves calling pending callbacks and clients
assume that their callback will only ever happen in softirq context.
Delay cleanup to the tasklet.
Signed-off-by: NDan Williams <dan.j.williams@intel.com>
Acked-by: NZhang Wei <wei.zhang@freescale.com>

The driver implements DMA engine API for Freescale MPC85xx DMA controller,
which could be used by devices in the silicon.  The driver supports the
Basic mode of Freescale MPC85xx DMA controller.  The MPC85xx processors
supported include MPC8540/60, MPC8555, MPC8548, MPC8641 and so on.

The MPC83xx(MPC8349, MPC8360) are also supported.

[kamalesh@linux.vnet.ibm.com: build fix]
[dan.j.williams@intel.com: merge mm fixes, rebase on async_tx-2.6.25]
Signed-off-by: NZhang Wei <wei.zhang@freescale.com>
Signed-off-by: NEbony Zhu <ebony.zhu@freescale.com>
Acked-by: NKumar Gala <galak@gate.crashing.org>
Cc: Shannon Nelson <shannon.nelson@intel.com>
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: Paul Mackerras <paulus@samba.org>
Signed-off-by: NAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>