Split the device_control callback of the Freescale Elo DMA driver to make use
of the newly introduced callbacks, that will eventually be used to retrieve
slave capabilities.

While we're at it, remove the useless prep_sg callback.
Signed-off-by: NMaxime Ripard <maxime.ripard@free-electrons.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

Split the device_control callback of the Moxart DMA driver to make use of the
newly introduced callbacks, that will eventually be used to retrieve slave
capabilities.
Signed-off-by: NMaxime Ripard <maxime.ripard@free-electrons.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

Split the device_control callback of the Marvell MMP TDMA driver to make use
of the newly introduced callbacks, that will eventually be used to retrieve
slave capabilities.
Signed-off-by: NMaxime Ripard <maxime.ripard@free-electrons.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

Split the device_control callback of the Marvell MMP PDMA driver to make use
of the newly introduced callbacks, that will eventually be used to retrieve
slave capabilities.
Signed-off-by: NMaxime Ripard <maxime.ripard@free-electrons.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

Split the device_control callback of the Hisilicon K3 DMA driver to make use
of the newly introduced callbacks, that will eventually be used to retrieve
slave capabilities.
Signed-off-by: NMaxime Ripard <maxime.ripard@free-electrons.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

Split the device_control callback of the IPU IDMAC driver to make use
of the newly introduced callbacks, that will eventually be used to retrieve
slave capabilities.
Signed-off-by: NMaxime Ripard <maxime.ripard@free-electrons.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

Split the device_control callback of the Intel MID DMA driver to make use
of the newly introduced callbacks, that will eventually be used to retrieve
slave capabilities.
Signed-off-by: NMaxime Ripard <maxime.ripard@free-electrons.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

Split the device_control callback of the Freescale IMX SDMA driver to make use
of the newly introduced callbacks, that will eventually be used to retrieve
slave capabilities.
Signed-off-by: NMaxime Ripard <maxime.ripard@free-electrons.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

Split the device_control callback of the Freescale IMX DMA driver to make use
of the newly introduced callbacks, that will eventually be used to retrieve
slave capabilities.
Signed-off-by: NMaxime Ripard <maxime.ripard@free-electrons.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

Split the device_control callback of the Freescale EDMA driver to make use
of the newly introduced callbacks, that will eventually be used to retrieve
slave capabilities.
Signed-off-by: NMaxime Ripard <maxime.ripard@free-electrons.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

Split the device_control callback of the Cirrus Logic EP93xx driver to make use
of the newly introduced callbacks, that will eventually be used to retrieve
slave capabilities.
Signed-off-by: NMaxime Ripard <maxime.ripard@free-electrons.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

Split the device_control callback of the TI EDMA driver to make use of the
newly introduced callbacks, that will eventually be used to retrieve slave
capabilities.
Signed-off-by: NMaxime Ripard <maxime.ripard@free-electrons.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

Split the device_control callback of the DesignWare DMA driver to make use
of the newly introduced callbacks, that will eventually be used to retrieve
slave capabilities.
Signed-off-by: NMaxime Ripard <maxime.ripard@free-electrons.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

Split the device_control callback of the JZ4740 DMA driver to make use of the
newly introduced callbacks, that will eventually be used to retrieve slave
capabilities.
Signed-off-by: NMaxime Ripard <maxime.ripard@free-electrons.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

Split the device_control callback of the TI CPPI41 DMA driver to make use
of the newly introduced callbacks, that will eventually be used to retrieve
slave capabilities.
Signed-off-by: NMaxime Ripard <maxime.ripard@free-electrons.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

Split the device_control callback of the ST-Ericsson COH901318 DMA driver to
make use of the newly introduced callbacks, that will eventually be used to
retrieve slave capabilities.
Signed-off-by: NMaxime Ripard <maxime.ripard@free-electrons.com>
Acked-by: NLinus Walleij <linus.walleij@linaro.org>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

Split the device_control callback of the Broadcom BCM2835 DMA driver to make
use of the newly introduced callbacks, that will eventually be used to retrieve
slave capabilities.
Signed-off-by: NMaxime Ripard <maxime.ripard@free-electrons.com>
Acked-by: NStephen Warren <swarren@wwwdotorg.org>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

Use newly introduced callbacks.
Signed-off-by: NLudovic Desroches <ludovic.desroches@atmel.com>
Signed-off-by: NMaxime Ripard <maxime.ripard@free-electrons.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

Split the device_control callback of the Atmel HDMAC driver to make use
of the newly introduced callbacks, that will eventually be used to retrieve
slave capabilities.
Signed-off-by: NMaxime Ripard <maxime.ripard@free-electrons.com>
Acked-by: NNicolas Ferre <nicolas.ferre@atmel.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

Split the device_control callback of the AMBA PL08x DMA driver to make use
of the newly introduced callbacks, that will eventually be used to retrieve
slave capabilities.
Signed-off-by: NMaxime Ripard <maxime.ripard@free-electrons.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

In order to migrate the drivers without triggering a BUG_ON for the converted
drivers, which would cause bisectability issues, we need to remove that check
before removing the device_control function entirely.
Signed-off-by: NMaxime Ripard <maxime.ripard@free-electrons.com>
Acked-by: NLaurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

Nowadays, some drivers don't have anything in there channel allocation
callbacks anymore.

Remove the BUG_ON if those callbacks aren't implemented, in order to allow
drivers to not implement them.
Signed-off-by: NMaxime Ripard <maxime.ripard@free-electrons.com>
Acked-by: NLaurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

dma_chan_get uses a rather interesting error handling and code path.

Change it to something more usual in the kernel.
Signed-off-by: NMaxime Ripard <maxime.ripard@free-electrons.com>
Acked-by: NLaurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

The dmaengine header abbreviates destination as at least two different strings.
Make a coherent use of a single one.
Signed-off-by: NMaxime Ripard <maxime.ripard@free-electrons.com>
Acked-by: NMark Brown <broonie@kernel.org>
Acked-by: NLaurent Pinchart <laurent.pinchart@ideasonboard.com>
Acked-by: NStephen Warren <swarren@wwwdotorg.org>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

The free_percpu() function tests whether its argument is NULL and then
returns immediately. Thus the test around the call is not needed.

This issue was detected by using the Coccinelle software.
Signed-off-by: NMarkus Elfring <elfring@users.sourceforge.net>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

Several systems are showing the following stack trace:

WARNING: CPU: 0 PID: 2352 at lib/dma-debug.c:1140 check_unmap+0x4ee/0x9e0()
ioatdma 0000:00:04.0: DMA-API: device driver failed to check map error[device address=0x0000000465bad000] [size=4096 bytes] [mapped as page]
Modules linked in: ioatdma(E+) nfsv3 rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache cfg80211 rfkill x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel cdc_ether ses ghash_clmulni_intel usbnet mii enclosure aesni_intel lrw gf128mul glue_helper iTCO_wdt shpchp ablk_helper iTCO_vendor_support cryptd pcspkr ipmi_devintf sb_edac lpc_ich edac_core mfd_core ipmi_si i2c_i801 wmi ipmi_msghandler nfsd auth_rpcgss nfs_acl lockd sunrpc xfs libcrc32c sd_mod crc_t10dif crct10dif_common mgag200 syscopyarea sysfillrect sysimgblt drm_kms_helper ttm igb drm ptp pps_core dca i2c_algo_bit i2ccore megaraid_sas dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ioatdma]
CPU: 0 PID: 2352 Comm: insmod Tainted: G            E  3.17.0-rc4+ #14
Hardware name: HP ProLiant m300 Server Cartridge/, BIOS H02 01/30/2014
 0000000000000009 ffff88007994b7d8 ffffffff816e7225 ffff88007994b820
 ffff88007994b810 ffffffff8107e51d ffff88045fc56c00 ffff88046643ee90
 ffffffff8338ccd0 0000000000000286 ffffffff81956629 ffff88007994b870
Call Trace:
 [<ffffffff816e7225>] dump_stack+0x4d/0x66
 [<ffffffff8107e51d>] warn_slowpath_common+0x7d/0xa0
 [<ffffffff8107e58c>] warn_slowpath_fmt+0x4c/0x50
 [<ffffffff81381e6e>] check_unmap+0x4ee/0x9e0
 [<ffffffff813823bf>] debug_dma_unmap_page+0x5f/0x70
 [<ffffffffa04546d8>] ioat_xor_val_self_test+0x498/0xcf0 [ioatdma]
 [<ffffffff81204f0a>] ? kfree+0xda/0x2b0
 [<ffffffffa044d510>] ? ioat_dma_setup_interrupts+0x120/0x2d0 [ioatdma]
 [<ffffffffa0454f4e>] ioat3_dma_self_test+0x1e/0x30 [ioatdma]
 [<ffffffffa044f904>] ioat_probe+0xf4/0x110 [ioatdma]
 [<ffffffffa04550f8>] ioat3_dma_probe+0x198/0x3a0 [ioatdma]
 [<ffffffffa044d18e>] ioat_pci_probe+0x11e/0x1b0 [ioatdma]
 [<ffffffff81393a15>] local_pci_probe+0x45/0xa0
 [<ffffffff81394be5>] ? pci_match_device+0xe5/0x110
 [<ffffffff81394d29>] pci_device_probe+0xd9/0x130
 [<ffffffff81462860>] driver_probe_device+0x90/0x3c0
 [<ffffffff81462c63>] __driver_attach+0x93/0xa0
 [<ffffffff81462bd0>] ? __device_attach+0x40/0x40
 [<ffffffff8146080b>] bus_for_each_dev+0x6b/0xb0
 [<ffffffff814622ce>] driver_attach+0x1e/0x20
 [<ffffffff81461ed8>] bus_add_driver+0x188/0x260
 [<ffffffffa0423000>] ? 0xffffffffa0423000
 [<ffffffff81463734>] driver_register+0x64/0xf0
 [<ffffffff813933a0>] __pci_register_driver+0x60/0x70
 [<ffffffffa0423089>] ioat_init_module+0x89/0x1000 [ioatdma]
 [<ffffffff8100212c>] do_one_initcall+0xbc/0x200
 [<ffffffff811e8b22>] ? __vunmap+0xd2/0x120
 [<ffffffff8111e73c>] load_module+0x14ec/0x1b50
 [<ffffffff81119970>] ? store_uevent+0x40/0x40
 [<ffffffff8111ef36>] SyS_finit_module+0x86/0xb0
 [<ffffffff816f1469>] system_call_fastpath+0x16/0x1b
---[ end trace 1052ccbbc3db4d08 ]---
Mapped at:
 [<ffffffff81380be1>] debug_dma_map_page+0x91/0x140
 [<ffffffffa045440e>] ioat_xor_val_self_test+0x1ce/0xcf0 [ioatdma]
 [<ffffffffa0454f4e>] ioat3_dma_self_test+0x1e/0x30 [ioatdma]
 [<ffffffffa044f904>] ioat_probe+0xf4/0x110 [ioatdma]
 [<ffffffffa04550f8>] ioat3_dma_probe+0x198/0x3a0 [ioatdma]

This happens because the current ioatdma DMA test code does not check the return
value of dma_map_page() calls with dma_mapping_error().  In addition, it was
noticed that mapping for the variable dest_dma is free'd before the last use.

This patch fixes these errors by initializing the dma_srcs[] array and checking
the returns with dma_mapping_error().
Signed-off-by: NPrarit Bhargava <prarit@redhat.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

The manual says that we need to (repeatedly) set the TearDown-bit for
the endpoint in order to get the active transfer descriptor released.
Doing this "real" quick over and over again seems to work but it also
seems that the hardware might not have enough time to breathe. So I
though, hey lets add a udelay() between between the individual sets
of the bit.
This change with the g_zero testcase resulted in a warning about missing
transfer descriptor (we got the tear-down one). It seems that if the
hardware has some time it manages to release the transfer-descriptor on
the completion queue after the teaddown descriptor.
With this change, I observe that the transfer descriptor is released
after 20-30 retry loops.
Signed-off-by: NSebastian Andrzej Siewior <bigeasy@linutronix.de>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

For a "complete" teardown we have to wait until the teardown descriptor
is returned by the hardware. The g_zero testcase "testusb -a -t 9" triggers
the following warning quite reliable:

|------------[ cut here ]------------
|WARNING: CPU: 0 PID: 0 at drivers/dma/cppi41.c:609 cppi41_dma_control+0x198/0x304()
|[<c003f84c>] (warn_slowpath_null) from [<c02be8d8>]
|[<c02be8d8>] (cppi41_dma_control) from [<bf08d25c>]
|[<bf08d25c>] (cppi41_dma_channel_abort [musb_hdrc])
|[<bf08bc38>] (nuke.constprop.10 [musb_hdrc])
|[<bf08bd08>] (musb_gadget_disable [musb_hdrc])
|[<bf252524>] (disable_endpoints [usb_f_ss_lb])
|[<bf2525d8>] (disable_source_sink [usb_f_ss_lb])
|[<bf25260c>] (sourcesink_set_alt [usb_f_ss_lb])
|[<bf23ad24>] (composite_setup [libcomposite])
|[<bf08a2f4>] (musb_g_ep0_irq [musb_hdrc])
|[<bf085ec4>] (musb_interrupt [musb_hdrc])
|[<bf0aeaf4>] (dsps_interrupt [musb_dsps])
|[<c0080ea8>] (handle_irq_event_percpu)
|[<c008112c>] (handle_irq_event)
|[<c008348c>] (handle_level_irq)
|[<c00807a8>] (generic_handle_irq)
|[<c000ee80>] (handle_IRQ)
|[<c00085f0>] (omap3_intc_handle_irq)

and complains about a TD descriptor which is not returned. I've been
looking at several things and haven't noticed anything unusual that
might lead to this.
The manual says "to try again" until the descriptor comes out. I limited
the amount of retries to 100 retries in order to avoid an infinite number
of retries and so a busy-loop. Back then testing revealed that the
number of retries were around 20-30 so 100 seemed a good upper limit.
This g_zero test reaches without a problem 98 retries and it jumps
sometimes to 101 on am335x-evm and so the WARN_ON() triggers. Same test
run on beaglebone black and the retries start at 122 and my max value so
far was at 128.
So lets rise the limit to 500.
Signed-off-by: NSebastian Andrzej Siewior <bigeasy@linutronix.de>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

The offset of all 8-/16-bit registers in big-endian eDMA model are
swapped in a 32-bit size opposite those in the little-endian model.

The hardware Scatter/Gather requires the subsequent TCDs stored in memory
in little endian independent of the register endian model, the eDMA engine
will do the swap if need.

This patch also use regular assignment for tcd variables r/w
instead of with io function previously that may not always be true.
Signed-off-by: NJingchang Lu <jingchang.lu@freescale.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

If the current transfer control descriptor (TCD) was not yet started,
the address will be the same as the initial address. Hence test if the
current address is less than or equal to the start address of each TCD.
Signed-off-by: NStefan Agner <stefan@agner.ch>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

After commit b2b49ccb (PM: Kconfig: Set PM_RUNTIME if PM_SLEEP is
selected) PM_RUNTIME is always set if PM is set, so #ifdef blocks
depending on CONFIG_PM_RUNTIME may now be changed to depend on
CONFIG_PM.

Replace CONFIG_PM_RUNTIME with CONFIG_PM in drivers/dma/nbpfaxi.c
and drivers/dma/tegra20-apb-dma.c.
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>
Acked-by: NVinod Koul <vinod.koul@intel.com>

pch_dma_id_table is used in pch_dma.c only, so declare
it as static
Signed-off-by: NMichele Curti <michele.curti@gmail.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

Return a negative error code on failure.

A simplified version of the semantic match that finds this problem is as
follows: (http://coccinelle.lip6.fr/)

// <smpl>
@@
identifier ret; expression e1,e2;
@@
(
if (\(ret < 0\|ret != 0\))
 { ... return ret; }
|
ret = 0
)
... when != ret = e1
    when != &ret
*if(...)
{
  ... when != ret = e2
      when forall
 return ret;
}
// </smpl>
Signed-off-by: NJulia Lawall <Julia.Lawall@lip6.fr>
Acked-by: NLinus Walleij <linus.walleij@linaro.org>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

When a firmware cannot be found for the SDMA engine then we can
continue with the internal ROM firmware.

The meaning of this message is frequently asked for, so make clear
that the driver still works with the internal ROM firmware and reduce
the loglevel from err to info.
Signed-off-by: NSascha Hauer <s.hauer@pengutronix.de>
Signed-off-by: NFabio Estevam <fabio.estevam@freescale.com>
Acked-by: NShawn Guo <shawn.guo@linaro.org>
Acked-by: NSascha Hauer <s.hauer@pengutronix.de>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

Update email-id to a personal one, as I have changed employment.
Signed-off-by: NJassi Brar <jassisinghbrar@gmail.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

The SET_PM_RUNTIME_PM_OPS() and SET_RUNTIME_PM_OPS() macros are
identical except that one of them is not empty for CONFIG_PM set,
while the other one is not empty for CONFIG_PM_RUNTIME set,
respectively.

However, after commit b2b49ccb (PM: Kconfig: Set PM_RUNTIME if
PM_SLEEP is selected) PM_RUNTIME is always set if PM is set, so one
of these macros is now redundant.

For this reason, replace SET_PM_RUNTIME_PM_OPS() with
SET_RUNTIME_PM_OPS() everywhere and redefine the SET_PM_RUNTIME_PM_OPS
symbol as SET_RUNTIME_PM_OPS in case new code is starting to use the
macro being removed here.
Reviewed-by: NUlf Hansson <ulf.hansson@linaro.org>
Acked-by: NKevin Hilman <khilman@linaro.org>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

Add system suspend/resume capabilities to the pl330 driver so the amba
bus clock could be also unprepared to conserve energy.
Signed-off-by: NKrzysztof Kozlowski <k.kozlowski@samsung.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

Signed-off-by: NRalf Baechle <ralf@linux-mips.org>

This patch adds runtime PM support to pl330 DMA engine driver.

The runtime power management for pl330 DMA driver allows gating of AMBA
clock (PDMA) in FSYS clock domain, when the device is not processing any
requests. This is necessary to enter low power modes on Exynos SoCs
(e.g. LPA on Exynos4x12 or W-AFTR on Exynos3250).

Runtime PM resuming of the device may happen in atomic context (during
call device_issue_pending()) so pm_runtime_irq_safe() is used. This will
lead only to disabling/enabling of the clock but this is sufficient for
gating the clock and for reducing energy usage.

Driver uses runtime PM callbacks from amba/bus.c driver only.
Suggested-by: NBartlomiej Zolnierkiewicz <b.zolnierkie@samsung.com>
Signed-off-by: NKrzysztof Kozlowski <k.kozlowski@samsung.com>
Reviewed-by: NUlf Hansson <ulf.hansson@linaro.org>
Acked-by: NVinod Koul <vinod.koul@intel.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

same issue as commit 7f5ae355:
"Without DMA_PRIVATE the driver is not able to allocate more than one channel.
Since it uses dma_get_any_slave_channel that calls private_candidate, the
second allocation fails at
/* some channels are already publicly allocated */
"
Signed-off-by: NLudovic Desroches <ludovic.desroches@atmel.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>