The number of bytes we want to fill into any LLI is the minimum of:
- number of bytes remaining in the transfer
- number of bytes we can transfer in a single LLI
- number of bytes we can transfer without overflowing the source boundary
- number of bytes we can transfer without overflowing the destination boundary

The minimum of the first two is already calculated (target_len).  We
limit the boundary calculations to this number of bytes, which will
then give us the number of bytes we can place into this LLI.
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>
Acked-by: NLinus Walleij <linus.walleij@stericsson.com>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>
Acked-by: NLinus Walleij <linus.walleij@stericsson.com>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

pl08x_pre_boundary() was unsafe with addresses towards the top of
memory space:

	boundary = ((addr >> PL08X_BOUNDARY_SHIFT) + 1)
			<< PL08X_BOUNDARY_SHIFT;

This can overflow a 32-bit number, producing zero.  When it does:

	if (boundary < addr + len)
		return boundary - addr;
	else
		return len;

results in (boundary - addr) returning either a large positive value.
Also if addr + len overflows, this calculation also fails.

We can fix this trivially as the only thing we're actually interested
in is the value of the least significant PL08X_BOUNDARY_SHIFT bits:

	boundary_len = PL08X_BOUNDARY_SIZE -
		(addr & (PL08X_BOUNDARY_SIZE - 1));

gives us the number of bytes before 'addr' becomes a multiple of
PL08X_BOUNDARY_SIZE.  We can then just take the min() of the two
calculated lengths.
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>
Acked-by: NLinus Walleij <linus.walleij@stericsson.com>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

We don't need pl08x_fill_lli_for_desc() to return num_llis + 1 as
we know that's what it always does.  We can just pass in num_llis
and use post-increment in the caller.

This makes the code slightly easier to read.
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>
Acked-by: NLinus Walleij <linus.walleij@stericsson.com>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

Calling the callback handler with spinlocks in the tasklet held leads
to deadlock when dmaengine functions are called:

BUG: spinlock lockup on CPU#0, sh/417, c1870a08
Backtrace:
...
[<c017b408>] (do_raw_spin_lock+0x0/0x154) from [<c02c4b98>] (_raw_spin_lock_irqsave+0x54/0x60)
[<c02c4b44>] (_raw_spin_lock_irqsave+0x0/0x60) from [<c01f5828>] (pl08x_prep_channel_resources+0x718/0x8b4)
[<c01f5110>] (pl08x_prep_channel_resources+0x0/0x8b4) from [<c01f5bb4>] (pl08x_prep_slave_sg+0x120/0x19c)
[<c01f5a94>] (pl08x_prep_slave_sg+0x0/0x19c) from [<c01be7a0>] (pl011_dma_tx_refill+0x164/0x224)
[<c01be63c>] (pl011_dma_tx_refill+0x0/0x224) from [<c01bf1c8>] (pl011_dma_tx_callback+0x7c/0xc4)
[<c01bf14c>] (pl011_dma_tx_callback+0x0/0xc4) from [<c01f4d34>] (pl08x_tasklet+0x60/0x368)
[<c01f4cd4>] (pl08x_tasklet+0x0/0x368) from [<c004d978>] (tasklet_action+0xa0/0x100)

Dan quoted the documentation:
> 2/ Completion callback routines cannot submit new operations.  This
>    results in recursion in the synchronous case and spin_locks being
>    acquired twice in the asynchronous case.

but then followed up to say:
> I should clarify, this is the async_memcpy() api requirement which is
> not used outside of md/raid5.  DMA drivers can and do allow new
> submissions from callbacks, and the ones that do so properly move the
> callback outside of the driver lock.

So let's fix it by moving the callback out of the spinlocked region.
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>
Acked-by: NLinus Walleij <linus.walleij@stericsson.com>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

Platforms need to be able to control which AHB master interface is used,
as each AHB master interface may be asymetric.  Allow the interfaces
used for fetching LLIs, memory, and each peripheral to be configured
individually.
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>
Acked-by: NLinus Walleij <linus.walleij@stericsson.com>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

As we initialize the default cctl value in the prep_* functions along
with the increment settings, we don't need to repeat the selection of
the AHB ports each time we create a LLI entry.  Do this in the prep_*
functions once per transfer.
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>
Acked-by: NLinus Walleij <linus.walleij@stericsson.com>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

We don't need to initialize the cctl increment and protection values
in the runtime_config method - we have all the inforamtion to setup
these values in prep_slave_sg().  Move their initialization there.
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>
Acked-by: NLinus Walleij <linus.walleij@stericsson.com>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

Rather than modifying platform data while preparing a transfer, copy
the cctl value into the txd structure and modify the value there.
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>
Acked-by: NLinus Walleij <linus.walleij@stericsson.com>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

There is no need to wait until we start processing a tx descriptor
before setting up the DMA request selection in the ccfg register.
We know which channel and request will be used in prep_phy_channel(),
so setup the ccfg request selection at txd creation time in
prep_phy_channel().
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>
Acked-by: NLinus Walleij <linus.walleij@stericsson.com>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

The ccfg register is used to configure the channel parameters - the type
and direction of transfer, the flow control signal and IRQ mask enables.
The type and direction of transfer is known in the relevent prep_*
function where a txd is created.  The IRQ mask enables are always set,
and the flow control signals are always set when we start processing a
txd according to phychan->signal.

If we store the ccfg value in the txd structure, we can avoid modifying
platform data - and even having it in platform data at all.

So, remove it from platform data too.
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>
Acked-by: NLinus Walleij <linus.walleij@stericsson.com>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

As we now have all the code accessing the phychan {csrc,cdst,clli,cctl,
ccfg} members in one function, there's no point storing the data into
the struct.  Get rid of the struct members.  Re-order the register dump
in the dev_dbg() to reflect the order we write the registers to the DMA
device.

The txd {csrc,cdst,clli,cctl} values are duplicates of the lli[0]
values, so there's no point duplicating these either.  Program the DMAC
registers directly from the lli[0] values.
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>
Acked-by: NLinus Walleij <linus.walleij@stericsson.com>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

There is no need for pl08x_config_phychan_for_txd(), pl08x_set_cregs()
and pl08x_enable_phy_chan() to be separate - they are always called in
sequence.  Combine them into one function.
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>
Acked-by: NLinus Walleij <linus.walleij@stericsson.com>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

As the LLI list is an array, we can use maths to locate which LLI
index we're currently at, and then sum up the remaining LLI entries
until we reach the end of the list.

This makes the code much easier to read, and much less susceptible
to falling off the end of the array.
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>
Acked-by: NLinus Walleij <linus.walleij@stericsson.com>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

The LLI pointer in the documentation is placed into the LLI register,
so name it LLI rather than 'next'.
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>
Acked-by: NLinus Walleij <linus.walleij@stericsson.com>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

Use 'u32' for the LLI structure members, which are defined by hardware
to be 32-bit.  dma_addr_t is much more vague about its actual size.
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>
Acked-by: NLinus Walleij <linus.walleij@stericsson.com>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

Use size_t for variables denoting lengths throughout, and use the 'z'
qualifier for printing the value.  For safety, add a BUG_ON() in
pl08x_fill_lli_for_desc() to catch the remainder potentially becoming
negative.
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>
Acked-by: NLinus Walleij <linus.walleij@stericsson.com>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

llis_bus is the DMA address of the LLI array.  Casting it to be a
pointer just to be able to use pointer arithmetic on it is not nice.
We can trivially deal with the places where we do arithmetic on it,
and it's actually cleaner this way.
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>
Acked-by: NLinus Walleij <linus.walleij@stericsson.com>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

We only want use the address of the LLI pointer when locating the
corresponding structure in memory, so clear the master bus selection
bit.
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>
Acked-by: NLinus Walleij <linus.walleij@stericsson.com>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

Tight loops should use cpu_relax() to allow CPUs to reduce power
consumption while waiting for events.
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>
Acked-by: NLinus Walleij <linus.walleij@stericsson.com>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

Consolidate duplicated channel release code into release_phy_channel()
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>
Acked-by: NLinus Walleij <linus.walleij@stericsson.com>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

Consolidate code which allocates and initializes txds.
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>
Acked-by: NLinus Walleij <linus.walleij@stericsson.com>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

Avoid using 'void *' struct fields when the structs are not defined
in linux/amba/pl08x.h - instead, forward declare the struct names, and
use these instead.  This ensures we have proper typechecking.
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>
Acked-by: NLinus Walleij <linus.walleij@stericsson.com>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

We should never modify the vendor data structure so make it const.
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>
Acked-by: NLinus Walleij <linus.walleij@stericsson.com>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

The driver already won't initialize a channel with a circular buffer;
the check in pl08x_prep_channel_resources() sees to that.  Remove
circular buffer support for the time being.
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>
Acked-by: NLinus Walleij <linus.walleij@stericsson.com>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

The tasklet always is initialized with a non-NULL data argument.  It
is not possible for it to be called with a NULL data argument (unless
something is very wrong in the tasklet code - in which case lots of
stuff will break).  Therefore, as plchan can never be NULL, remove
this unnecessary BUG check.

In pl08x_tasklet(), we've already dereferenced plchan->at, so it can't
be NULL here.  Remove this unnecessary BUG check.

pl08x_fill_llis_for_desc() and pl08x_free_txd() are always called with
a non-NULL txd argument - either as a consequence of the code paths or
as a result of other checks already in place.  We don't need to repeat
the non-NULL check in these functions.
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>
Acked-by: NLinus Walleij <linus.walleij@stericsson.com>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

We don't need to include linux/pci.h as we aren't a PCI driver.  We
aren't doing any processor specific functions, so asm/processor.h is
not required.  asm/cacheflush.h shouldn't be used, we have the DMA API
for this.  DMA interfaces aren't required as we're only implementing
the dmaengine API and not a platform-private DMA API.
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>
Acked-by: NLinus Walleij <linus.walleij@stericsson.com>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

A driver which emits both decimal and hex numbers in its printk
creates confusion as to what is what.  Prefix hex numbers with 0x.
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>
Acked-by: NLinus Walleij <linus.walleij@stericsson.com>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

Include the revision number of the PL08x primecell in the boot-time
printk to allow proper identification of the peripheral.  Reformat
the announcement printk format reflect what we do for other primecell
drivers - generally "PLXXX revX at 0xNNNNNNNN irq X".
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>
Acked-by: NLinus Walleij <linus.walleij@stericsson.com>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

Explain the two flow control methods which the PL08x implements, along
with the problem which peripheral flow control presents.  This helps
people understand why we are unable to use these DMA controllers with
(eg) the MMCI.
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>
Acked-by: NLinus Walleij <linus.walleij@stericsson.com>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

drivers/dma/amba-pl08x.c:1895:40: warning: Unknown escape '%'
drivers/dma/amba-pl08x.c:1903:40: warning: Unknown escape '%'
drivers/dma/amba-pl08x.c:513:6: warning: symbol 'pl08x_choose_master_bus' was not declared. Should it be static?
drivers/dma/amba-pl08x.c:604:5: warning: symbol 'pl08x_fill_llis_for_desc' was not declared. Should it be static?
drivers/dma/amba-pl08x.c:1442:32: warning: symbol 'pl08x_prep_slave_sg' was not declared. Should it be static?
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>
Acked-by: NLinus Walleij <linus.walleij@stericsson.com>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

Trying to disable a tasklet while holding a spinlock which the tasklet
will take is a recipe for deadlock - tasklet_disable() will wait for the
tasklet to finish running, which it will never do.  In any case, there
is not a corresponding tasklet_enable(), so once the tasklet is disabled,
it will never run again until reboot.

It's safe to just remove the tasklet_disable() as we remove all current
and pending descriptors before releasing this spinlock.  This means that
the tasklet will find no remaining work if it subsequently runs.

The only remaining issue is that the callback for an already submitted
txd may be in progress, or even called after terminate_all() returns.
There's not much that can be done about that as waiting for the callback
to complete before returning will also lead to deadlocks.
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>
Acked-by: NLinus Walleij <linus.walleij@stericsson.com>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

pl08x_issue_pending() returns with the spinlock locked and interrupts
disabled if the channel is waiting for a physical DMA to become free.
This is wrong - especially as pl08x_issue_pending() is an API function
as it leads to deadlocks.  Fix it to always return with the spinlock
unlocked.
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>
Acked-by: NLinus Walleij <linus.walleij@stericsson.com>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

If we fail to allocate the LLI, the prep_* function will return NULL.
However, the TXD we allocated will not be placed on any list, nor
will it be freed - we'll just drop all references to it.  Make sure
we free it rather than leaking TXDs.
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>
Acked-by: NLinus Walleij <linus.walleij@stericsson.com>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

Tasklets are run from an interruptible context.  The slave DMA functions
can be called from within IRQ handlers.  Taking the spinlock without
disabling interrupts allows an interrupt handler to run, which may try
to take the spinlock again, resulting in deadlock.  Fix this by using
the irqsave spinlocks.
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>
Acked-by: NLinus Walleij <linus.walleij@stericsson.com>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

The last_issued variable uses an atomic type, which is only
incremented inside a protected region, and then read.  Everywhere else
only reads the value, so it isn't using atomic_t correctly, and it
doesn't even need to.  Moreover, the DMA engine code provides us with
a variable for this already - chan.cookie.  Use chan.cookie instead.

Also, avoid negative dma_cookie_t values - negative returns from
tx_submit() mean failure, yet in reality we always succeed.  Restart
from cookie 1, just like other DMA engine drivers do.
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>
Acked-by: NLinus Walleij <linus.walleij@stericsson.com>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

If maxburst was passed in as zero, we would overflow the burst_sizes[]
array.  Fix this by checking for this condition, and defaulting to
single transfer 'bursts'.

Improve the readability of the loop using a for() loop rather than
a while() loop with the iterator initialized far from the loop.
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>
Acked-by: NLinus Walleij <linus.walleij@stericsson.com>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

Correct mis-spellings in comments and printk strings.
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>
Acked-by: NLinus Walleij <linus.walleij@stericsson.com>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

* 'merge' of git://git.kernel.org/pub/scm/linux/kernel/git/benh/powerpc:
  powerpc: Use call_rcu_sched() for pagetables