This patch plumbs the existing ARM IOMMU DMA infrastructure (which isn't
actually called outside of a few drivers) into arch_setup_dma_ops, so
that we can use IOMMUs for DMA transfers in a more generic fashion.

Since this significantly complicates the arch_setup_dma_ops function,
it is moved out of line into dma-mapping.c. If CONFIG_ARM_DMA_USE_IOMMU
is not set, the iommu parameter is ignored and the normal ops are used
instead.
Acked-by: NRussell King <rmk+kernel@arm.linux.org.uk>
Acked-by: NArnd Bergmann <arnd@arndb.de>
Acked-by: NMarek Szyprowski <m.szyprowski@samsung.com>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

This patch extends of_dma_configure so that it sets up the IOMMU for a
device, as well as the coherent/non-coherent DMA mapping ops.
Acked-by: NArnd Bergmann <arnd@arndb.de>
Acked-by: NMarek Szyprowski <m.szyprowski@samsung.com>
Tested-by: NRobin Murphy <robin.murphy@arm.com>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

set_arch_dma_coherent_ops is called from of_dma_configure in order to
swizzle the architectural dma-mapping functions over to a cache-coherent
implementation. This is currently implemented only for ARM.

In anticipation of re-using this mechanism for IOMMU-backed dma-mapping
ops too, this patch replaces the function with a broader
arch_setup_dma_ops callback which will be extended in future.
Acked-by: NArnd Bergmann <arnd@arndb.de>
Acked-by: NMarek Szyprowski <m.szyprowski@samsung.com>
Tested-by: NRobin Murphy <robin.murphy@arm.com>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

Provide an implementation for dma_{alloc,free,mmap}_writecombine() when
the architecture supports DMA attributes.
Signed-off-by: NThierry Reding <treding@nvidia.com>
Acked-by: NArnd Bergmann <arnd@arndb.de>
Signed-off-by: NMarek Szyprowski <m.szyprowski@samsung.com>

Implement the set_arch_dma_coherent_ops() for ARM architecture.

Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Cc: Russell King <linux@arm.linux.org.uk>
Cc: Arnd Bergmann <arnd@arndb.de>
Cc: Olof Johansson <olof@lixom.net>
Cc: Grant Likely <grant.likely@linaro.org>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Linus Walleij <linus.walleij@linaro.org>
Reviewed-by: NRob Herring <robh@kernel.org>
Signed-off-by: NGrygorii Strashko <grygorii.strashko@ti.com>
Signed-off-by: NSantosh Shilimkar <santosh.shilimkar@ti.com>

In most of cases DMA addresses can be performed using offset value of
 Bus address space relatively to physical address space as following:

PFN->DMA:
 __pfn_to_phys(pfn + [-]dma_pfn_offset)

DMA->PFN:
 __phys_to_pfn(dma_addr) + [-]dma_pfn_offset

Thanks to Russell King for suggesting the optimised macro's for
conversion.

Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Cc: Russell King <linux@arm.linux.org.uk>
Cc: Arnd Bergmann <arnd@arndb.de>
Cc: Olof Johansson <olof@lixom.net>
Cc: Grant Likely <grant.likely@linaro.org>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Linus Walleij <linus.walleij@linaro.org>
Reviewed-by: NRob Herring <robh@kernel.org>
Signed-off-by: NGrygorii Strashko <grygorii.strashko@ti.com>
Signed-off-by: NSantosh Shilimkar <santosh.shilimkar@ti.com>

IOMMU_HELPER is needed because SWIOTLB calls iommu_is_span_boundary,
provided by lib/iommu_helper.c.
Signed-off-by: NStefano Stabellini <stefano.stabellini@eu.citrix.com>
Reviewed-by: NKonrad Rzeszutek Wilk <konrad.wilk@oracle.com>
CC: will.deacon@arm.com
Acked-by: NRussell King <rmk+kernel@arm.linux.org.uk>

Changes in v9:
- remove uneeded include asm/cacheflush.h;
- just return 0 if !dev->dma_mask in dma_capable.

Changes in v8:
- use __phys_to_pfn and __pfn_to_phys.

Changes in v7:
- dma_mark_clean: empty implementation;
- in dma_capable use coherent_dma_mask if dma_mask hasn't been
  allocated.

Changes in v6:
- check for dev->dma_mask being NULL in dma_capable.

Changes in v5:
- implement dma_mark_clean using dmac_flush_range.

Changes in v3:
- dma_capable: do not treat dma_mask as a limit;
- remove SWIOTLB dependency on NEED_SG_DMA_LENGTH.

Most of the kernel code assumes that max*pfn is maximum pfns because
the physical start of memory is expected to be PFN0. Since this
assumption is not true on ARM architectures, the meaning of max*pfn
is number of memory pages. This is done to keep drivers happy which
are making use of of these variable to calculate the dma bounce limit
using dma_mask.

Now since we have a architecture override possibility for DMAable
maximum pfns, lets make meaning of max*pfns as maximum pnfs on ARM
as well.

Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Will Deacon <will.deacon@arm.com>
Cc: Nicolas Pitre <nicolas.pitre@linaro.org>
Signed-off-by: NSantosh Shilimkar <santosh.shilimkar@ti.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

We can't simply override arm_dma_ops with xen_dma_ops because devices
are allowed to have their own dma_ops and they take precedence over
arm_dma_ops. When running on Xen as initial domain, we always want
xen_dma_ops to be the one in use.

We introduce __generic_dma_ops to allow xen_dma_ops functions to call
back to the native implementation.
Signed-off-by: NStefano Stabellini <stefano.stabellini@eu.citrix.com>
Suggested-by: NCatalin Marinas <catalin.marinas@arm.com>
Acked-by: NRussell King <rmk+kernel@arm.linux.org.uk>
CC: will.deacon@arm.com
CC: linux@arm.linux.org.uk

Changes in v7:
- return xen_dma_ops only if we are the initial domain;
- rename __get_dma_ops to __generic_dma_ops.

IOMMU_HELPER is needed because SWIOTLB calls iommu_is_span_boundary,
provided by lib/iommu_helper.c.
Signed-off-by: NStefano Stabellini <stefano.stabellini@eu.citrix.com>
Reviewed-by: NKonrad Rzeszutek Wilk <konrad.wilk@oracle.com>
CC: will.deacon@arm.com
CC: linux@arm.linux.org.uk


Changes in v8:
- use __phys_to_pfn and __pfn_to_phys.

Changes in v7:
- dma_mark_clean: empty implementation;
- in dma_capable use coherent_dma_mask if dma_mask hasn't been
  allocated.

Changes in v6:
- check for dev->dma_mask being NULL in dma_capable.

Changes in v5:
- implement dma_mark_clean using dmac_flush_range.

Changes in v3:
- dma_capable: do not treat dma_mask as a limit;
- remove SWIOTLB dependency on NEED_SG_DMA_LENGTH.

Without the patch, kind of below warning will be dumped if DMA-API
debug is enabled:

[   11.069763] ------------[ cut here ]------------
[   11.074645] WARNING: at lib/dma-debug.c:948 check_unmap+0x770/0x860()
[   11.081420] ehci-omap ehci-omap.0: DMA-API: device driver failed to
check map error[device address=0x0000000
0adb78e80] [size=8 bytes] [mapped as single]
[   11.095611] Modules linked in:

Cc: Russell King <linux@arm.linux.org.uk>
Cc: Marek Szyprowski <m.szyprowski@samsung.com>
Signed-off-by: NMing Lei <ming.lei@canonical.com>
Signed-off-by: NJoerg Roedel <joro@8bytes.org>

Expose another DMA operations function: arm_dma_set_mask. This
function will be added to a custom DMA ops for Armada 370/XP.
Depending of its configuration Armada 370/XP can be set as a "nearly"
coherent architecture. In this case the DMA ops is made of:
- specific functions for this architecture
- already exposed arm DMA related functions
- the arm_dma_set_mask which was not exposed yet.
Signed-off-by: NGregory CLEMENT <gregory.clement@free-electrons.com>
Acked-by: NMarek Szyprowski <m.szyprowski@samsung.com>

Since commit e9da6e99 ("ARM: dma-mapping: remove custom consistent
dma region") setting consistent dma memory size is not longer required. All
calls to this function has been already removed, so the
init_consistent_dma_size() stub can also be gone.
Signed-off-by: NMarek Szyprowski <m.szyprowski@samsung.com>

This reverts commit 871ae57a, which is
scheduled for v3.8 and accidently got into v3.7-rc series.
Signed-off-by: NMarek Szyprowski <m.szyprowski@samsung.com>

Without the patch, kind of below warning will be dumped if DMA-API
debug is enabled:

[   11.069763] ------------[ cut here ]------------
[   11.074645] WARNING: at lib/dma-debug.c:948 check_unmap+0x770/0x860()
[   11.081420] ehci-omap ehci-omap.0: DMA-API: device driver failed to
check map error[device address=0x0000000
0adb78e80] [size=8 bytes] [mapped as single]
[   11.095611] Modules linked in:

Cc: Russell King <linux@arm.linux.org.uk>
Cc: Marek Szyprowski <m.szyprowski@samsung.com>
Signed-off-by: NMing Lei <ming.lei@canonical.com>
Signed-off-by: NMarek Szyprowski <m.szyprowski@samsung.com>

arch_is_coherent is problematic as it is a global symbol. This
doesn't work for multi-platform kernels or platforms which can support
per device coherent DMA.

This adds arm_coherent_dma_ops to be used for devices which connected
coherently (i.e. to the ACP port on Cortex-A9 or A15). The arm_dma_ops
are modified at boot when arch_is_coherent is true.
Signed-off-by: NRob Herring <rob.herring@calxeda.com>
Cc: Russell King <linux@arm.linux.org.uk>
Cc: Marek Szyprowski <m.szyprowski@samsung.com>
Signed-off-by: NMarek Szyprowski <m.szyprowski@samsung.com>

Some platforms might require to increase atomic coherent pool to make
sure that their device will be able to allocate all their buffers from
atomic context. This function can be also used to decrease atomic
coherent pool size if coherent allocations are not used for the given
sub-platform.
Suggested-by: NJosh Coombs <josh.coombs@gmail.com>
Signed-off-by: NMarek Szyprowski <m.szyprowski@samsung.com>

This patch adds support for dma_get_sgtable() function which is required
to let drivers to share the buffers allocated by DMA-mapping subsystem.

Generic implementation based on virt_to_page() is not suitable for ARM
dma-mapping subsystem.
Signed-off-by: NMarek Szyprowski <m.szyprowski@samsung.com>
Reviewed-by: NKyungmin Park <kyungmin.park@samsung.com>

Commit 9adc5374 ('common: dma-mapping: introduce mmap method') added a
generic method for implementing mmap user call to dma_map_ops structure.

This patch converts ARM and PowerPC architectures (the only providers of
dma_mmap_coherent/dma_mmap_writecombine calls) to use this generic
dma_map_ops based call and adds a generic cross architecture
definition for dma_mmap_attrs, dma_mmap_coherent, dma_mmap_writecombine
functions.

The generic mmap virt_to_page-based fallback implementation is provided for
architectures which don't provide their own implementation for mmap method.
Signed-off-by: NMarek Szyprowski <m.szyprowski@samsung.com>
Reviewed-by: NKyungmin Park <kyungmin.park@samsung.com>

This patch changes dma-mapping subsystem to use generic vmalloc areas
for all consistent dma allocations. This increases the total size limit
of the consistent allocations and removes platform hacks and a lot of
duplicated code.

Atomic allocations are served from special pool preallocated on boot,
because vmalloc areas cannot be reliably created in atomic context.
Signed-off-by: NMarek Szyprowski <m.szyprowski@samsung.com>
Reviewed-by: NKyungmin Park <kyungmin.park@samsung.com>
Reviewed-by: NMinchan Kim <minchan@kernel.org>

This patch converts dma_alloc/free/mmap_{coherent,writecombine}
functions to use generic alloc/free/mmap methods from dma_map_ops
structure. A new DMA_ATTR_WRITE_COMBINE DMA attribute have been
introduced to implement writecombine methods.
Signed-off-by: NMarek Szyprowski <m.szyprowski@samsung.com>
Acked-by: NKyungmin Park <kyungmin.park@samsung.com>
Acked-by: NArnd Bergmann <arnd@arndb.de>
Tested-By: NSubash Patel <subash.ramaswamy@linaro.org>

This patch removes dma bounce hooks from the common dma mapping
implementation on ARM architecture and creates a separate set of
dma_map_ops for dma bounce devices.
Signed-off-by: NMarek Szyprowski <m.szyprowski@samsung.com>
Acked-by: NKyungmin Park <kyungmin.park@samsung.com>
Tested-By: NSubash Patel <subash.ramaswamy@linaro.org>

This patch modifies dma-mapping implementation on ARM architecture to
use common dma_map_ops structure and asm-generic/dma-mapping-common.h
helpers.
Signed-off-by: NMarek Szyprowski <m.szyprowski@samsung.com>
Acked-by: NKyungmin Park <kyungmin.park@samsung.com>
Tested-By: NSubash Patel <subash.ramaswamy@linaro.org>

This patch removes the need for the offset parameter in dma bounce
functions. This is required to let dma-mapping framework on ARM
architecture to use common, generic dma_map_ops based dma-mapping
helpers.

Background and more detailed explaination:

dma_*_range_* functions are available from the early days of the dma
mapping api. They are the correct way of doing a partial syncs on the
buffer (usually used by the network device drivers). This patch changes
only the internal implementation of the dma bounce functions to let
them tunnel through dma_map_ops structure. The driver api stays
unchanged, so driver are obliged to call dma_*_range_* functions to
keep code clean and easy to understand.

The only drawback from this patch is reduced detection of the dma api
abuse. Let us consider the following code:

dma_addr = dma_map_single(dev, ptr, 64, DMA_TO_DEVICE);
dma_sync_single_range_for_cpu(dev, dma_addr+16, 0, 32, DMA_TO_DEVICE);

Without the patch such code fails, because dma bounce code is unable
to find the bounce buffer for the given dma_address. After the patch
the above sync call will be equivalent to:

dma_sync_single_range_for_cpu(dev, dma_addr, 16, 32, DMA_TO_DEVICE);

which succeeds.

I don't consider this as a real problem, because DMA API abuse should be
caught by debug_dma_* function family. This patch lets us to simplify
the internal low-level implementation without chaning the driver visible
API.
Signed-off-by: NMarek Szyprowski <m.szyprowski@samsung.com>
Acked-by: NKyungmin Park <kyungmin.park@samsung.com>
Tested-By: NSubash Patel <subash.ramaswamy@linaro.org>

Replace all uses of ~0 with DMA_ERROR_CODE, what should make the code
easier to read.
Signed-off-by: NMarek Szyprowski <m.szyprowski@samsung.com>
Acked-by: NKyungmin Park <kyungmin.park@samsung.com>
Tested-By: NSubash Patel <subash.ramaswamy@linaro.org>

This is to avoid a compiler warning when invoking the __bus_to_virt()
macro. The dma_to_virt() function gets addresses within the 32-bit
range.
Signed-off-by: NCatalin Marinas <catalin.marinas@arm.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

This function can be called during boot to increase the size of the consistent
DMA region above it's default value of 2MB. It must be called before the memory
allocator is initialised, i.e. before any core_initcall.
Signed-off-by: NJon Medhurst <tixy@yxit.co.uk>
Acked-by: NNicolas Pitre <nicolas.pitre@linaro.org>

ISA_DMA_THRESHOLD has been unused by non-arch code, so lets now get
rid of it from ARM by replacing it with arm_dma_zone_mask.  Move
dma_supported() and dma_set_mask() out of line, and have
dma_supported() check this new variable instead.
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

Simplify the dmabounce specific code in dma_set_mask().  We can just
omit setting the dma mask if dmabounce is enabled (we will have already
set dma mask via callbacks when the device is created in that case.)
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

Pass the device type specific needs_bounce function in at dmabounce
register time, avoiding the need for a platform specific global
function to do this.
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

Use dma_map_page()/dma_unmap_page() internals to handle dma_map_single()
and dma_unmap_single().
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

Add ARM support for the DMA debug infrastructure, which allows the
DMA API usage to be debugged.
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

Replace the page_to_dma() and dma_to_page() macros with their PFN
equivalents.  This allows us to map parts of memory which do not have
a struct page allocated to them to bus addresses.  This will be used
internally by dma_alloc_coherent()/dma_alloc_writecombine().

Build tested on Versatile, OMAP1, IOP13xx and KS8695.
Tested-by: NJanusz Krzysztofik <jkrzyszt@tis.icnet.pl>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

This reverts commit 4fa5518c, which causes a compilation regression for
IXP4xx platforms.
Reported-by: NRichard Cochran <richardcochran@gmail.com>
Acked-by: NEric Miao <eric.y.miao@gmail.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

Architectures implement dma_is_consistent() in different ways (some
misinterpret the definition of API in DMA-API.txt).  So it hasn't been so
useful for drivers.  We have only one user of the API in tree.  Unlikely
out-of-tree drivers use the API.

Even if we fix dma_is_consistent() in some architectures, it doesn't look
useful at all.  It was invented long ago for some old systems that can't
allocate coherent memory at all.  It's better to export only APIs that are
definitely necessary for drivers.

Let's remove this API.
Signed-off-by: NFUJITA Tomonori <fujita.tomonori@lab.ntt.co.jp>
Cc: James Bottomley <James.Bottomley@HansenPartnership.com>
Reviewed-by: NKonrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Cc: <linux-arch@vger.kernel.org>
Signed-off-by: NAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>

dma_get_cache_alignment returns the minimum DMA alignment.  Architectures
defines it as ARCH_DMA_MINALIGN (formally ARCH_KMALLOC_MINALIGN).  So we
can unify dma_get_cache_alignment implementations.

Note that some architectures implement dma_get_cache_alignment wrongly.
dma_get_cache_alignment() should return the minimum DMA alignment.  So
fully-coherent architectures should return 1.  This patch also fixes this
issue.
Signed-off-by: NFUJITA Tomonori <fujita.tomonori@lab.ntt.co.jp>
Cc: <linux-arch@vger.kernel.org>
Signed-off-by: NAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>

With a correct dev->dma_mask before calling dmabounce_register_dev(),
dma_needs_bounce() is not necessary.

The sa1111, though, is a bit complicated. Until it's fully understood
and fixed, dma_needs_bounce() for sa1111 is kept if CONFIG_SA1111 is
enabled with no side effect (with the condition of machine_is_*)

Thanks for Mike Rapoport to fix one error in the original version of
the patch and get this tested.
Acked-by: NMike Rapoport <mike@compulab.co.il>
Signed-off-by: NEric Miao <eric.y.miao@gmail.com>

This converts arm to the generic pci_set_dma_mask and
pci_set_consistent_dma_mask (removes HAVE_ARCH_PCI_SET_DMA_MASK for
dmabounce).
Signed-off-by: NFUJITA Tomonori <fujita.tomonori@lab.ntt.co.jp>
Looked-over-by: NRussell King <rmk+kernel@arm.linux.org.uk>
Cc: Jesse Barnes <jbarnes@virtuousgeek.org>
Cc: Greg KH <greg@kroah.com>
Cc: Kay Sievers <kay.sievers@vrfy.org>
Signed-off-by: NAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>

Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>
Tested-By: NSantosh Shilimkar <santosh.shilimkar@ti.com>

The DMA API has the notion of buffer ownership; make it explicit in the
ARM implementation of this API.  This gives us a set of hooks to allow
us to deal with CPU cache issues arising from non-cache coherent DMA.
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>
Tested-By: NSantosh Shilimkar <santosh.shilimkar@ti.com>
Tested-By: NJamie Iles <jamie@jamieiles.com>