Move all DMA descriptor prepping functions to prep.c file. Fixup all
broken bits caused by the move.
Signed-off-by: NDave Jiang <dave.jiang@intel.com>
Acked-by: NDan Williams <dan.j.williams@intel.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

Moving all the init routines to init.c and fixup anything broken during
the move.
Signed-off-by: NDave Jiang <dave.jiang@intel.com>
Acked-by: NDan Williams <dan.j.williams@intel.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

Move and fixup all sysfs related bits to sysfs.c file.
Signed-off-by: NDave Jiang <dave.jiang@intel.com>
Acked-by: NDan Williams <dan.j.williams@intel.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

Clean out dma_v2 and remove ioat2 calls since we are moving everything
to just ioat.
Signed-off-by: NDave Jiang <dave.jiang@intel.com>
Acked-by: NDan Williams <dan.j.williams@intel.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

Changing the variable names for ioatdma_device to be consistently named
ioat_dma instead of device/dma in order to avoid confusion and distinct
from struct device. This will clearly indicate that it is an
ioatdma_device. This also make all the naming consistent that the dma
device is ioat_dma and all the channels are ioat_chan.
Signed-off-by: NDave Jiang <dave.jiang@intel.com>
Acked-by: NDan Williams <dan.j.williams@intel.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

Kill the common ioatdma channel structure and everything that is not
dma_chan to be ioat_dma_chan. Since we don't have to worry about v1
and v2 ioatdma anymore this makes it much cleaner and obvious for
maintenance.
Signed-off-by: NDave Jiang <dave.jiang@intel.com>
Acked-by: NDan Williams <dan.j.williams@intel.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

Removal of support for ioatdma v2 device support.
Signed-off-by: NDave Jiang <dave.jiang@intel.com>
Acked-by: NDan Williams <dan.j.williams@intel.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

Cleaning up of ioat1 specific code as it is no longer supported
Signed-off-by: NDave Jiang <dave.jiang@intel.com>
Acked-by: NDan Williams <dan.j.williams@intel.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

Removal of any devices that are ioatdma pre-3.0. This is the first step
in attempting to clean up the ioatdma driver and remove hw no longer
supported.
Signed-off-by: NDave Jiang <dave.jiang@intel.com>
Acked-by: NDan Williams <dan.j.williams@intel.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

If the allocation order is 16, then the u16 count will overflow and wrap
to zero when assigned the value 1 << 16.

Change the type of 'total_descs' to int, so that it is large enough to
store a value equal or greater than 1 << 16.
Signed-off-by: NAllen Hubbe <Allen.Hubbe@emc.com>
Acked-by: NDave Jiang <dave.jiang@intel.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

If the allocation order is 16, then the u16 index will overflow and wrap
to zero instead of being equal or greater than 1 << 16.  The loop
condition will always be true, and the loop will run until all the
memory resources are depleted.

Change the type of index 'i' to u32, so that it is large enough to store
a value equal or greater than 1 << 16.
Signed-off-by: NAllen Hubbe <Allen.Hubbe@emc.com>
Acked-by: NDave Jiang <dave.jiang@intel.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

The functions irq_irq_err and ipu_irq_fn are identical plus/minus the
comments. Remove one.
Signed-off-by: NThomas Gleixner <tglx@linutronix.de>
Cc: Vinod Koul <vinod.koul@intel.com>
Cc: Dan Williams <dan.j.williams@intel.com>
Cc: dmaengine@vger.kernel.org
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

The irq argument of most interrupt flow handlers is unused or merily
used instead of a local variable. The handlers which need the irq
argument can retrieve the irq number from the irq descriptor.

Search and update was done with coccinelle and the invaluable help of
Julia Lawall.
Signed-off-by: NThomas Gleixner <tglx@linutronix.de>
Cc: Julia Lawall <Julia.Lawall@lip6.fr>
Cc: Vinod Koul <vinod.koul@intel.com>
Cc: Dan Williams <dan.j.williams@intel.com>
Cc: dmaengine@vger.kernel.org
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

The XDMAC also supports memset operations over discontiguous areas. Add the
necessary logic to support this.
Signed-off-by: NMaxime Ripard <maxime.ripard@free-electrons.com>
Acked-by: NLudovic Desroches <ludovic.desroches@atmel.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

Most drivers need to set constraints on the buffer alignment for async tx
operations. However, even though it is documented, some drivers either use
a defined constant that is not matching what the alignment variable expects
(like DMA_BUSWIDTH_* constants) or fill the alignment in bytes instead of
power of two.

Add a new enum for these alignments that matches what the framework
expects, and convert the drivers to it.
Signed-off-by: NMaxime Ripard <maxime.ripard@free-electrons.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

This patch adds DEV_TO_DEV support for i.MX SDMA driver to support data
transfer between two peripheral FIFOs.
The per_2_per script requires two peripheral addresses and two DMA
requests, and it need to check the src addr and dst addr is in the SPBA
bus space or in the AIPS bus space.
Signed-off-by: NShengjiu Wang <shengjiu.wang@freescale.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

Use irq_desc_get_xxx() to avoid redundant lookup of irq_desc while we
already have a pointer to corresponding irq_desc.

This is also a preparation for the removal of the 'irq' argument from
interrupt flow handlers.
Signed-off-by: NJiang Liu <jiang.liu@linux.intel.com>
Cc: Dan Williams <dan.j.williams@intel.com>
Cc: Vinod Koul <vinod.koul@intel.com>
Cc: dmaengine@vger.kernel.org
Signed-off-by: NThomas Gleixner <tglx@linutronix.de>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

Chained irq handlers usually set up handler data as well. We now have
a function to set both under irq_desc->lock. Replace the two calls
with one.

Search and conversion was done with coccinelle.
Reported-by: NRussell King <rmk+kernel@arm.linux.org.uk>
Signed-off-by: NThomas Gleixner <tglx@linutronix.de>
Cc: Julia Lawall <Julia.Lawall@lip6.fr>
Cc: Dan Williams <dan.j.williams@intel.com>
Cc: Vinod Koul <vinod.koul@intel.com>
Cc: dmaengine@vger.kernel.org
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

All hardware accesses are done under virtual channel lock. That's why specific
channel lock is excessive and can be removed safely. This has been tested on
Intel Medfield and Merrifield.
Signed-off-by: NAndy Shevchenko <andriy.shevchenko@linux.intel.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

The dmaengine core assumes that async DMA devices will only be removed
when they not used anymore, or it assumes dma_async_device_unregister()
will only be called by dma driver exit routines. But this assumption is
not true for the IOAT driver, which calls dma_async_device_unregister()
from ioat_remove(). So current IOAT driver doesn't support device
hot-removal because it may cause system crash to hot-remove an inuse
IOAT device.

To support CPU socket hot-removal, all PCI devices, including IOAT
devices embedded in the socket, will be hot-removed. The idea solution
is to enhance the dmaengine core and IOAT driver to support hot-removal,
but that's too hard.

This patch implements a hack to disable IOAT devices under hotplug-capable
CPU socket so it won't break socket hot-removal.
Signed-off-by: NJiang Liu <jiang.liu@linux.intel.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

This allows claiming of non-RAID channels as a private channel. This
prevents breakage of MDRAID using the IOATDMA channels via
async_tx but also allows agents such as NTB to claim channels
exclusively for its usages.
Signed-off-by: NDave Jiang <dave.jiang@intel.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

Removing static analysis error:-
Possible null pointer dereference: xt

Because currently xt is dereferenced before NULL check,
Thus Use it after NULL Check.
Signed-off-by: NManinder Singh <maninder1.s@samsung.com>
Reviewed-by: NVaneet Narang <v.narang@samsung.com>
Acked-by: NNicolas Ferre <nicolas.ferre@atmel.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

clk_prepare_enable() may fail, so we should better check its return value
and propagate it in the case of error.

While at it, change the label 'err' to a more descriptive naming.
Signed-off-by: NFabio Estevam <fabio.estevam@freescale.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

Commit 3b62286d ("dmaengine: Remove FSF mailing addresses") left Free
Software Foundation mailing address still in two files. Remove it now.
Signed-off-by: NJarkko Nikula <jarkko.nikula@linux.intel.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

To be consistent with other kernel interface namings, rename
of_get_named_gen_pool() to of_gen_pool_get().  In the original function
name "_named" suffix references to a device tree property, which contains
a phandle to a device and the corresponding device driver is assumed to
register a gen_pool object.

Due to a weak relation and to avoid any confusion (e.g.  in future
possible scenario if gen_pool objects are named) the suffix is removed.

[sfr@canb.auug.org.au: crypto/marvell/cesa - fix up for of_get_named_gen_pool() rename]
Signed-off-by: NVladimir Zapolskiy <vladimir_zapolskiy@mentor.com>
Cc: Nicolas Ferre <nicolas.ferre@atmel.com>
Cc: Philipp Zabel <p.zabel@pengutronix.de>
Cc: Shawn Guo <shawn.guo@linaro.org>
Cc: Sascha Hauer <kernel@pengutronix.de>
Cc: Alexandre Belloni <alexandre.belloni@free-electrons.com>
Cc: Russell King <linux@arm.linux.org.uk>
Cc: Mauro Carvalho Chehab <mchehab@osg.samsung.com>
Cc: Vinod Koul <vinod.koul@intel.com>
Cc: Takashi Iwai <tiwai@suse.de>
Cc: Jaroslav Kysela <perex@perex.cz>
Signed-off-by: NStephen Rothwell <sfr@canb.auug.org.au>
Cc: Herbert Xu <herbert@gondor.apana.org.au>
Cc: Boris BREZILLON <boris.brezillon@free-electrons.com>
Signed-off-by: NAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>

drivers/dma/xgene-dma.c has file permissions 775, which is wrong, it should
be 664, so fix it
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

Clear pending interrupts before requesting interrupts and move
interrupt initialization after channels have been initialized.
This avoids a NULL pointer dereference panic when using kexec
while DMA requests were running.
Signed-off-by: NStefan Agner <stefan@agner.ch>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

The XDMAC supports memset transfers, both over contiguous areas, and over
discontiguous areas through a LLI.

The current memset operation only supports contiguous memset for now, add some
support for it. Scatter-gathered memset will come eventually.
Signed-off-by: NMaxime Ripard <maxime.ripard@free-electrons.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

This patch attempts to enhance the case of a transfer submitted multiple
times, and where the cost of creating the descriptors chain is not
negligible.

This happens with big video buffers (several megabytes, ie. several
thousands of linked descriptors in one scatter-gather list). In these
cases, a video driver would want to do :
 - tx = dmaengine_prep_slave_sg()
 - dma_engine_submit(tx);
 - dma_async_issue_pending()
 - wait for video completion
 - read video data (or not, skipping a frame is also possible)
 - dma_engine_submit(tx)
   => here, the descriptors chain recalculation will take time
   => the dma coherent allocation over and over might create holes in
      the dma pool, which is counter-productive.
 - dma_async_issue_pending()
 - etc ...

In order to cope with this case, virt-dma is modified to prevent freeing
the descriptors upon completion if DMA_CTRL_ACK flag is set in the
transfer.
Signed-off-by: NRobert Jarzmik <robert.jarzmik@free.fr>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

This reverts commit 48a9db46.

Some platforms actually need support for the memset operations. Bring it back.
Signed-off-by: NMaxime Ripard <maxime.ripard@free-electrons.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

The AT91 HDMAC controller supports interleaved transfers through what's
called the Picture-in-Picture mode, which allows to transfer a squared
portion of a framebuffer.

This means that this interleaved transfer only supports interleaved
transfers which have a transfer size and ICGs that are fixed across all the
chunks.

While this is a quite drastic restriction of the interleaved transfers
compared to what the dmaengine API allows, this is still useful, and our
driver will only reject transfers that do not conform to this.
Signed-off-by: NMaxime Ripard <maxime.ripard@free-electrons.com>
Acked-by: NLudovic Desroches <ludovic.desroches@atmel.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

Now that we can have ICGs set for both the source and destination (using
the icg field of struct data_chunk) or for only the source or the
destination (using the dst_icg or src_icg respectively), and that these
fields can be ignored depending on other parameters (src_inc, src_sgl,
etc.), the logic to get the actual ICG value can be quite tricky.

The XDMAC driver was already implementing it, but since we will need it in
other drivers, we can move it to the main header file.
Signed-off-by: NMaxime Ripard <maxime.ripard@free-electrons.com>
Acked-by: NLudovic Desroches <ludovic.desroches@atmel.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

Some drivers implement only pause operation (no resuming). Example is
pl330 where pause is needed for getting residuum. pl330 does not support
resume operation, transfer must be stopped after pause.

However for slaves this is exposed always as "pause and resume" which
introduces subtle errors on Odroid U3 board (Exynos4412 with pl330).
After adding pause function to pl330 driver the audio playback
(utilizing DMA) gets choppy after some time (approximately 24 hours).

Fix this by exposing "cmd_pause" if and only if pause and resume are
implemented.
Signed-off-by: NKrzysztof Kozlowski <k.kozlowski@samsung.com>
Reported-by: gabriel@unseen.is
Reported-by: NMarek Szyprowski <m.szyprowski@samsung.com>
Cc: <stable@vger.kernel.org>
Fixes: 88987d2c ("dmaengine: pl330: add DMA_PAUSE feature")
Acked-by: NMaxime Ripard <maxime.ripard@free-electrons.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

This patch change the way free descriptors are marked.

Instead of having a field for descriptor in use, all the descriptors in the
all_slots list are free for use.

This simplify the allocation method and reduce the locking needed.
Signed-off-by: NLior Amsalem <alior@marvell.com>
Reviewed-by: NOfer Heifetz <oferh@marvell.com>
Signed-off-by: NMaxime Ripard <maxime.ripard@free-electrons.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

Now that we have 2 channels assigned to 2 CPUs and all requests are chained
on same channels, we need much more descriptors available to satisfy
async_tx workload.

3072 descriptors was found in our lab as the number of descriptors which
allow the async_tx stack to work without waiting for free descriptors on
submission of new requests.
Signed-off-by: NLior Amsalem <alior@marvell.com>
Reviewed-by: NNadav Haklai <nadavh@marvell.com>
Tested-by: NNadav Haklai <nadavh@marvell.com>
Signed-off-by: NMaxime Ripard <maxime.ripard@free-electrons.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

The Marvell Armada 38x SoC introduce new features to the XOR engine,
especially the fact that the engine mode (MEMCPY/XOR/PQ/etc) can be part of
the descriptor and not set through the controller registers.

This new feature allows mixing of different commands (even PQ) on the same
channel/chain without the need to stop the engine to reconfigure the engine
mode.

Refactor the driver to be able to use that new feature on the Armada 38x,
while keeping the old behaviour on the older SoCs.
Signed-off-by: NLior Amsalem <alior@marvell.com>
Reviewed-by: NOfer Heifetz <oferh@marvell.com>
Signed-off-by: NMaxime Ripard <maxime.ripard@free-electrons.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

The current function names isn't very consistent, and functions with the
same prefix might operate on either a channel or a descriptor, which is
kind of confusing.

Rename these functions to have a consistent and clearer naming scheme.
Signed-off-by: NMaxime Ripard <maxime.ripard@free-electrons.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

This patch fixes a bug in the XOR driver where the cleanup function can be
called and free descriptors that never been processed by the engine (which
result in data errors).

The cleanup function will free descriptors based on the ownership bit in
the descriptors.

Fixes: ff7b0479 ("dmaengine: DMA engine driver for Marvell XOR engine")
Signed-off-by: NLior Amsalem <alior@marvell.com>
Signed-off-by: NMaxime Ripard <maxime.ripard@free-electrons.com>
Reviewed-by: NOfer Heifetz <oferh@marvell.com>
Cc: stable@vger.kernel.org
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

The kernel is not trying to increase mcbufsz. It suggests you should try
doing so. Also print the calculated required size of mcbufsz.
Signed-off-by: NMichal Suchanek <hramrach@gmail.com>
Reviewed-by: NKrzysztof Kozlowski <k.kozlowski@samsung.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

Rework slave configuration part in order to more report wrong errors
about the configuration.
Only maxburst and addr width values are checked when doing the slave
configuration. The validity of the channel configuration is done at
prepare time.
Signed-off-by: NLudovic Desroches <ludovic.desroches@atmel.com>
Cc: stable@vger.kernel.org # 4.0 and later
Signed-off-by: NVinod Koul <vinod.koul@intel.com>