Cookies corresponding to pending transfers have a residue value equal to
the full size of the corresponding descriptor. The driver miscomputes
that and uses the size of the active descriptor instead. Fix it.
Reported-by: NGeert Uytterhoeven <geert@linux-m68k.org>
Signed-off-by: NLaurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
[geert: Also check desc.active list]
Signed-off-by: NGeert Uytterhoeven <geert+renesas@glider.be>
Signed-off-by: NNiklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

Running descriptor pointer is set to NULL upon freeing resources. Other-
wise, rcar_dmac_issue_pending might not start new transfers
Signed-off-by: NMuhammad Hamza Farooq <mfarooq@visteon.com>
Signed-off-by: NGeert Uytterhoeven <geert+renesas@glider.be>
Signed-off-by: NNiklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
Acked-by: NLaurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

The documentation states one should make sure both DE and TE are cleared
before starting a transaction. This patch extends the current warning to
look at both DE and TE.

Based on previous work from Muhammad Hamza Farooq.
Suggested-by: NMuhammad Hamza Farooq <mfarooq@visteon.com>
Signed-off-by: NNiklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
Acked-by: NLaurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

The hardware might have complete the transfer but the interrupt handler
might not have had a chance to run. If rcar_dmac_chan_get_residue()
which reads HW registers finds that there is no residue return
DMA_COMPLETE.
Signed-off-by: NMuhammad Hamza Farooq <mfarooq@visteon.com>
Signed-off-by: NGeert Uytterhoeven <geert+renesas@glider.be>
[Niklas: add explanation in commit message]
Signed-off-by: NNiklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
Acked-by: NLaurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

For each descriptor, in addition to the memory used by the descriptors
structure itself, the driver allocates a list of chunks as well as a
buffer for hardware descriptors. Descriptors themselves are preallocated,
and allocation of the chunks and buffer is performed the first time the
descriptor is used. The memory isn't freed when the transfer is completed,
as the chunks and buffer will be needed again when the descriptor is
reused internally, so the driver keeps the memory around.

If only a few descriptors are used concurrently, the current
list_add_tail() implementation will result in all preallocated descriptors
being used before going back to the first one, and will thus allocate
chunks and a buffer for all preallocated descriptors. Using list_add()
will put the complete descriptor at the head of the list of available
descriptors, so the next transfer will be more likely to reuse a
descriptor that already has associated memory instead of one that has
never been used before.
Signed-off-by: NKuninori Morimoto <kuninori.morimoto.gx@renesas.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

DMACHCLR clears each channels, but its channel number is based on
its SoC or IP. Current driver is using fixed 0x7fff (= for 15ch),
it is not good match for Gen3 or Gen2 Audio DMAC. This patch fixes it
Signed-off-by: NKuninori Morimoto <kuninori.morimoto.gx@renesas.com>
Acked-by: NGeert Uytterhoeven <geert+renesas@glider.be>
Acked-by: NLaurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

Use the generic mechanism to declare a bitmap instead of unsigned long.
Signed-off-by: NJoe Perches <joe@perches.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

Current rcar-dmac driver is using spin_lock_irq() / spin_unlock_irq()
in some functions. But, some other driver might call DMAEngine API
during interrupt disabled. In such case, rcar-dmac side spin_unlock_irq()
forcefully allows all interrupts. Therefore, other driver receives
unexpected interruption, and its exclusive access control will be broken.
This patch replaces spin_lock_irq() to spin_lock_irqsave(),
and spin_unlock_irq() to spin_unlock_irqrestore().
Reported-by: NCao Minh Hiep <cm-hiep@jinso.co.jp>
Signed-off-by: NKuninori Morimoto <kuninori.morimoto.gx@renesas.com>
Tested-by: NKeita Kobayashi <keita.kobayashi.ym@renesas.com>
Acked-by: NVinod Koul <vinod.koul@intel.com>
Signed-off-by: NMark Brown <broonie@kernel.org>

A still unconfirmed hardware bug prevents the IPMMU microTLB 0 to be
flushed correctly, resulting in memory corruption. DMAC 0 channel 0 is
connected to microTLB 0 on currently supported platforms, so we can't
use it with the IPMMU. As the IOMMU API operates at the device level we
can't disable it selectively, so ignore channel 0 for now if the device
is part of an IOMMU group.
Signed-off-by: NLaurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
Acked-by: NGeert Uytterhoeven <geert+renesas@glider.be>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

When descriptor memory is accessed through an IOMMU the DMADAR register
isn't initialized automatically from the first descriptor at beginning
of transfer by the DMAC like it should. Initialize it manually with the
destination address of the first chunk.
Signed-off-by: NLaurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
Acked-by: NGeert Uytterhoeven <geert+renesas@glider.be>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

When wired to an IOMMU to access data, the DMAC accesses the hardware
descriptors through the IOMMU as well. We're using the DMA mapping API
to allocate the descriptors, but with a NULL device at the moment, which
prevents IOMMU mappings from being created. Fix this by passing the DMAC
device instead.
Signed-off-by: NLaurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
Acked-by: NGeert Uytterhoeven <geert+renesas@glider.be>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

The error interrupt handler stops and reinitializes all channels. This
causes a crash for channels that have never been used, as their
descriptor lists are uninitialized. Fix it by initializing the
descriptor lists at probe time.
Signed-off-by: NLaurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
Acked-by: NGeert Uytterhoeven <geert+renesas@glider.be>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

The rcar_dmac_desc_put() function is called in interrupt context and
must thus use spin_lock_irqsave() instead of spin_lock_irq().
Signed-off-by: NLaurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
Acked-by: NGeert Uytterhoeven <geert+renesas@glider.be>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

The desc variable is used uninitialized in the rcar_dmac_desc_get() and
rcar_dmac_xfer_chunk_get() functions if descriptors need to be
allocated. Fix it.
Reported-by: NDan Carpenter <dan.carpenter@oracle.com>
Signed-off-by: NLaurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
Signed-off-by: NVinod Koul <vinod.koul@intel.com>

If the atomic DMA coherent pool is too small, disable use of hardware
descriptor lists instead of crashing the system:

ERROR: 256 KiB atomic DMA coherent pool is too small!
Please increase it with coherent_pool= kernel parameter!

Unable to handle kernel NULL pointer dereference at virtual address 00000004
Internal error: Oops: a07 [#1] PREEMPT SMP ARM

PC is at rcar_dmac_chan_reinit+0x3c/0x160
LR is at _raw_spin_lock_irqsave+0x18/0x5c

[<802132c0>] (rcar_dmac_chan_reinit) from [<80214818>] (rcar_dmac_isr_error+0x84/0xa0)
[<80214818>] (rcar_dmac_isr_error) from [<80060484>] (handle_irq_event_percpu+0x50/0x150)
[<80060484>] (handle_irq_event_percpu) from [<800605c0>] (handle_irq_event+0x3c/0x5c)
[<800605c0>] (handle_irq_event) from [<8006350c>] (handle_fasteoi_irq+0xb8/0x198)
[<8006350c>] (handle_fasteoi_irq) from [<8005fdb0>] (generic_handle_irq+0x20/0x30)
[<8005fdb0>] (generic_handle_irq) from [<8000fcd0>] (handle_IRQ+0x50/0xc4)
[<8000fcd0>] (handle_IRQ) from [<800092cc>] (gic_handle_irq+0x28/0x5c)
[<800092cc>] (gic_handle_irq) from [<80012700>] (__irq_svc+0x40/0x70)

Kernel panic - not syncing: Fatal exception in interrupt
Signed-off-by: NJürg Billeter <j@bitron.ch>
Signed-off-by: NLaurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>

Unlike DMA transfers descriptors that are preallocated and cached,
memory used to store hardware descriptors is allocated and freed with
the DMA coherent allocation API for every transfer. Besides degrading
performances, this creates a CMA stress test that seems to cause issues.
Running dmatest with the noverify option produces

[   50.066539] alloc_contig_range test_pages_isolated(6b845, 6b846) failed
[   50.235180] alloc_contig_range test_pages_isolated(6b848, 6b84e) failed
[   52.964584] alloc_contig_range test_pages_isolated(6b847, 6b848) failed
[   54.127113] alloc_contig_range test_pages_isolated(6b843, 6b844) failed
[   56.270253] alloc_contig_range test_pages_isolated(6b84c, 6b850) failed

The root cause needs to be fixed, but in the meantime, as a workaround
and a performance improvement, cache hardware descriptors.
Signed-off-by: NLaurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
Tested-by: NWolfram Sang <wsa+renesas@sang-engineering.com>

The DMAC supports hardware-based auto-configuration from descriptor
lists. This reduces the number of interrupts required for processing a
DMA transfer. Support that mode in the driver.
Signed-off-by: NLaurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
Tested-by: NWolfram Sang <wsa+renesas@sang-engineering.com>

The DMAC is a general purpose multi-channel DMA controller that supports
both slave and memcpy transfers.

The driver currently supports the DMAC found in the r8a7790 and r8a7791
SoCs. Support for compatible DMA controllers (such as the audio DMAC)
will be added later.

Feature-wise, automatic hardware handling of descriptors chains isn't
supported yet. LPAE support is implemented.
Signed-off-by: NLaurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
Tested-by: NKuninori Morimoto <kuninori.morimoto.gx@renesas.com>
Tested-by: NWolfram Sang <wsa+renesas@sang-engineering.com>