Map the buffer once and use dma_sync*() appropriately instead of mapping the
buffer over and over without unmapping it.
Acked-by: NPer Forlin <per.forlin@stericsson.com>
Acked-by: NJonas Aaberg <jonas.aberg@stericsson.com>
Signed-off-by: NRabin Vincent <rabin.vincent@stericsson.com>
Signed-off-by: NLinus Walleij <linus.walleij@stericsson.com>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

The address to use for DMA should be taken from sg_dma_address() and not
sg_phys().
Acked-by: NPer Forlin <per.forlin@stericsson.com>
Acked-by: NJonas Aaberg <jonas.aberg@stericsson.com>
Signed-off-by: NRabin Vincent <rabin.vincent@stericsson.com>
Signed-off-by: NLinus Walleij <linus.walleij@stericsson.com>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

ALIGN(x * y, y) == x * y
ALIGN(aligned + x * y, y) == aligned + x * y
Acked-by: NPer Forlin <per.forlin@stericsson.com>
Acked-by: NJonas Aaberg <jonas.aberg@stericsson.com>
Signed-off-by: NRabin Vincent <rabin.vincent@stericsson.com>
Signed-off-by: NLinus Walleij <linus.walleij@stericsson.com>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

DB8500v2's DMA40 (revision 3) allows setting event lines as high priority and
real time.
Acked-by: NPer Forlin <per.forlin@stericsson.com>
Acked-by: NJonas Aaberg <jonas.aberg@stericsson.com>
Signed-off-by: NRabin Vincent <rabin.vincent@stericsson.com>
Signed-off-by: NLinus Walleij <linus.walleij@stericsson.com>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

And add DB8500v2 information.
Acked-by: NPer Forlin <per.forlin@stericsson.com>
Acked-by: NJonas Aaberg <jonas.aberg@stericsson.com>
Signed-off-by: NRabin Vincent <rabin.vincent@stericsson.com>
Signed-off-by: NLinus Walleij <linus.walleij@stericsson.com>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

Almost every use of dev_err in this driver prints the function name.  Abstract
out wrappers to help with this and reduce code duplication.
Acked-by: NPer Forlin <per.forlin@stericsson.com>
Acked-by: NJonas Aaberg <jonas.aberg@stericsson.com>
Signed-off-by: NRabin Vincent <rabin.vincent@stericsson.com>
Signed-off-by: NLinus Walleij <linus.walleij@stericsson.com>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

The somewhat confusing check

	d40c->log_num == D40_PHY_CHAN

and its variants are used in several places to check if a channel is logical or
physical.  Use appropriately named helpers to do this to make the code more
readable.
Acked-by: NPer Forlin <per.forlin@stericsson.com>
Acked-by: NJonas Aaberg <jonas.aberg@stericsson.com>
Signed-off-by: NRabin Vincent <rabin.vincent@stericsson.com>
Signed-off-by: NLinus Walleij <linus.walleij@stericsson.com>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

The register offset computation for accessing channel registers is copy/pasted
in several places.  Create a helper function to do it.
Acked-by: NPer Forlin <per.forlin@stericsson.com>
Acked-by: NJonas Aaberg <jonas.aberg@stericsson.com>
Signed-off-by: NRabin Vincent <rabin.vincent@stericsson.com>
Signed-off-by: NLinus Walleij <linus.walleij@stericsson.com>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

This link in hardware with previous jobs code is:

  - unused, no clients using or requiring this feature
  - incomplete, being implemented only for physical channels
  - broken, only working to perform one link

Remove it.  This also allows us to get rid of the channel pause in the
submit_tx() routine.
Acked-by: NPer Forlin <per.forlin@stericsson.com>
Acked-by: NJonas Aaberg <jonas.aberg@stericsson.com>
Signed-off-by: NRabin Vincent <rabin.vincent@stericsson.com>
Signed-off-by: NLinus Walleij <linus.walleij@stericsson.com>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

The controller sometimes fails to register the enable of the event line when
both src and dst event lines are used on the same logical channel.  Implement
the recommended software workaround, which is to retry the write until it
works.
Acked-by: NPer Forlin <per.forlin@stericsson.com>
Acked-by: NJonas Aaberg <jonas.aberg@stericsson.com>
Signed-off-by: NRabin Vincent <rabin.vincent@stericsson.com>
Signed-off-by: NLinus Walleij <linus.walleij@stericsson.com>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

Acked-by: NPer Forlin <per.forlin@stericsson.com>
Acked-by: NJonas Aaberg <jonas.aberg@stericsson.com>
Signed-off-by: NRabin Vincent <rabin.vincent@stericsson.com>
Signed-off-by: NLinus Walleij <linus.walleij@stericsson.com>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

Cleanup the formatting of comments, remove some which don't make sense
anymore.
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>
[fix conflict with 96a608a4]
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

Commit f5e70d0f renamed MD_RAID6_PQ to RAID6_PQ,
but iop-adma.c didn't update synchronously.
Signed-off-by: NWei Yongquan <weiyqlq@gmail.com>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

We were reading channel status then taking a lock. This lead to a race because
this lock may delay us and then make this channel not idle anymore.
Signed-off-by: NNicolas Ferre <nicolas.ferre@atmel.com>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

Place flags on first descriptor of chain instead of last.
This is the one used by atc_chain_complete() function while unmapping.
Signed-off-by: NNicolas Ferre <nicolas.ferre@atmel.com>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

Use subsys_initcall instead of module_init in order to keep DMA engine rolling
before other peripheral drivers.
Signed-off-by: NEric Xu <hong.xu@atmel.com>
Signed-off-by: NNicolas Ferre <nicolas.ferre@atmel.com>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

Following descriptor flow in at_hdmac driver, descriptor comming from
atc_desc_get() as already DMA_CTRL_ACK flag set. No need to set it again.
Signed-off-by: NNicolas Ferre <nicolas.ferre@atmel.com>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

Signed-off-by: NNicolas Ferre <nicolas.ferre@atmel.com>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

In atc_prep_slave_sg() function we use dma_address field of scatterlist with
sg_dma_address() macro instead of sg_phys(). DMA address is already computed
by dma_map_sg() or another mapping function in calling driver.
Signed-off-by: NNicolas Ferre <nicolas.ferre@atmel.com>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

Support new device OKI SEMICONDUCTOR's ML7213 IOH(Input/Output Hub) which is for
IVI(In-Vehicle Infotainment) use.
The ML7213 is companion chip for Intel Atom E6xx series.
The ML7213 is completely compatible for Intel EG20T PCH.
Signed-off-by: NTomoya MORINAGA <tomoya-linux@dsn.okisemi.com>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

A recent patch has added a die notifier to the shdma driver, however,
it registers a static die-notifier object in the probe routine, i.e.,
for each device instance. This is wrong and leads to a system lockup.
This patch moves the die notifier registration to module init and
exit routines respectively.
Reported-by: NMagnus Damm <damm@opensource.se>
Signed-off-by: NGuennadi Liakhovetski <g.liakhovetski@gmx.de>
Signed-off-by: NPaul Mundt <lethal@linux-sh.org>

Prevent dma_set_runtime_config() being used to alter the configuration
supplied by the platform for memcpy channel configuration.  No one
should be trying to change this configuration.
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>
Acked-by: NLinus Walleij <linus.walleij@stericsson.com>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

There are cases in dma_set_runtime_config() where we fail to perform
the requested action - and we just issue a KERN_ERR message in that
case.  We have the facility to return an error to the caller, so that
is what we should do.

When we encounter an error due to invalid parameters, we should not
modify driver state.
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>
Acked-by: NLinus Walleij <linus.walleij@stericsson.com>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

The PL08x driver holds on to the channel lock with interrupts disabled
between the prepare and the subsequent submit API functions.  This
means that the locking state when the prepare function returns is
dependent on whether it suceeeds or not.

It did this to ensure that the physical channel wasn't released, and
as it used to add the descriptor onto the pending list at prepare time
rather than submit time.

Now that we have reorganized the code to remove those reasons, we can
now safely release the spinlock at the end of preparation and reacquire
it in our submit function.
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>
Acked-by: NLinus Walleij <linus.walleij@stericsson.com>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

Introduce 'phychan_hold' to hold on to physical DMA channels while we're
preparing a new descriptor for it.  This will be incremented when we
allocate a physical channel and set the MUX registers during the
preparation of the TXD, and will only be decremented when the TXD is
submitted.

This prevents the physical channel being given up before the new TXD
is placed on the queue.
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>
Acked-by: NLinus Walleij <linus.walleij@stericsson.com>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

Don't place TXDs on the pending list when they're prepared - place
them on the list when they're ready to be submitted.  Also, only
place memcpy requests in the wait state when they're submitted and
don't have a physical channel associated.
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>
Acked-by: NLinus Walleij <linus.walleij@stericsson.com>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

This 'desc_list' is actually a list of pending descriptors, so name
it after its function (pending list) rather than what it contains
(descriptors).
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>
Acked-by: NLinus Walleij <linus.walleij@stericsson.com>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

The DMA engine API requires DMA engine implementations to unmap buffers
passed into the non-slave DMA methods unless the relevant completion
flag is set.  We aren't doing this, so implement this facility.
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>
Acked-by: NLinus Walleij <linus.walleij@stericsson.com>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

Like other DMA engine drivers do, store the passed flags into the
async_tx structure, so they can be checked when the operation
completes.
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>
Acked-by: NLinus Walleij <linus.walleij@stericsson.com>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

We only need to store the dma address.
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>
Acked-by: NLinus Walleij <linus.walleij@stericsson.com>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

Don't alter any txd->srcbus or txd->dstbus values while building the
LLI list.  This allows us to see the original dma_addr_t values passed
in via the prep_memcpy() method.
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>
Acked-by: NLinus Walleij <linus.walleij@stericsson.com>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

The number of bytes we want to fill into any LLI is the minimum of:
- number of bytes remaining in the transfer
- number of bytes we can transfer in a single LLI
- number of bytes we can transfer without overflowing the source boundary
- number of bytes we can transfer without overflowing the destination boundary

The minimum of the first two is already calculated (target_len).  We
limit the boundary calculations to this number of bytes, which will
then give us the number of bytes we can place into this LLI.
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>
Acked-by: NLinus Walleij <linus.walleij@stericsson.com>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>
Acked-by: NLinus Walleij <linus.walleij@stericsson.com>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

pl08x_pre_boundary() was unsafe with addresses towards the top of
memory space:

	boundary = ((addr >> PL08X_BOUNDARY_SHIFT) + 1)
			<< PL08X_BOUNDARY_SHIFT;

This can overflow a 32-bit number, producing zero.  When it does:

	if (boundary < addr + len)
		return boundary - addr;
	else
		return len;

results in (boundary - addr) returning either a large positive value.
Also if addr + len overflows, this calculation also fails.

We can fix this trivially as the only thing we're actually interested
in is the value of the least significant PL08X_BOUNDARY_SHIFT bits:

	boundary_len = PL08X_BOUNDARY_SIZE -
		(addr & (PL08X_BOUNDARY_SIZE - 1));

gives us the number of bytes before 'addr' becomes a multiple of
PL08X_BOUNDARY_SIZE.  We can then just take the min() of the two
calculated lengths.
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>
Acked-by: NLinus Walleij <linus.walleij@stericsson.com>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

We don't need pl08x_fill_lli_for_desc() to return num_llis + 1 as
we know that's what it always does.  We can just pass in num_llis
and use post-increment in the caller.

This makes the code slightly easier to read.
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>
Acked-by: NLinus Walleij <linus.walleij@stericsson.com>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

Calling the callback handler with spinlocks in the tasklet held leads
to deadlock when dmaengine functions are called:

BUG: spinlock lockup on CPU#0, sh/417, c1870a08
Backtrace:
...
[<c017b408>] (do_raw_spin_lock+0x0/0x154) from [<c02c4b98>] (_raw_spin_lock_irqsave+0x54/0x60)
[<c02c4b44>] (_raw_spin_lock_irqsave+0x0/0x60) from [<c01f5828>] (pl08x_prep_channel_resources+0x718/0x8b4)
[<c01f5110>] (pl08x_prep_channel_resources+0x0/0x8b4) from [<c01f5bb4>] (pl08x_prep_slave_sg+0x120/0x19c)
[<c01f5a94>] (pl08x_prep_slave_sg+0x0/0x19c) from [<c01be7a0>] (pl011_dma_tx_refill+0x164/0x224)
[<c01be63c>] (pl011_dma_tx_refill+0x0/0x224) from [<c01bf1c8>] (pl011_dma_tx_callback+0x7c/0xc4)
[<c01bf14c>] (pl011_dma_tx_callback+0x0/0xc4) from [<c01f4d34>] (pl08x_tasklet+0x60/0x368)
[<c01f4cd4>] (pl08x_tasklet+0x0/0x368) from [<c004d978>] (tasklet_action+0xa0/0x100)

Dan quoted the documentation:
> 2/ Completion callback routines cannot submit new operations.  This
>    results in recursion in the synchronous case and spin_locks being
>    acquired twice in the asynchronous case.

but then followed up to say:
> I should clarify, this is the async_memcpy() api requirement which is
> not used outside of md/raid5.  DMA drivers can and do allow new
> submissions from callbacks, and the ones that do so properly move the
> callback outside of the driver lock.

So let's fix it by moving the callback out of the spinlocked region.
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>
Acked-by: NLinus Walleij <linus.walleij@stericsson.com>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

Platforms need to be able to control which AHB master interface is used,
as each AHB master interface may be asymetric.  Allow the interfaces
used for fetching LLIs, memory, and each peripheral to be configured
individually.
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>
Acked-by: NLinus Walleij <linus.walleij@stericsson.com>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

As we initialize the default cctl value in the prep_* functions along
with the increment settings, we don't need to repeat the selection of
the AHB ports each time we create a LLI entry.  Do this in the prep_*
functions once per transfer.
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>
Acked-by: NLinus Walleij <linus.walleij@stericsson.com>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

We don't need to initialize the cctl increment and protection values
in the runtime_config method - we have all the inforamtion to setup
these values in prep_slave_sg().  Move their initialization there.
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>
Acked-by: NLinus Walleij <linus.walleij@stericsson.com>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

Rather than modifying platform data while preparing a transfer, copy
the cctl value into the txd structure and modify the value there.
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>
Acked-by: NLinus Walleij <linus.walleij@stericsson.com>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>