All these symbols are only used by arch dma_ops implementations or
xen-swiotlb.  None of which can be modular.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Acked-by: NChristian König <christian.koenig@amd.com>

Factor out a new swiotlb_alloc_buffer helper that allocates DMA coherent
memory from the swiotlb bounce buffer.

This allows to simplify the swiotlb_alloc implemenation that uses
dma_direct_alloc to try to allocate a reachable buffer first.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Acked-by: NChristian König <christian.koenig@amd.com>

Factor out a new swiotlb_free_buffer helper that checks if an address
is allocated from the swiotlb bounce buffer, and if yes frees it.

This allows to simplify the swiotlb_free implemenation that uses
dma_direct_free to free the non-bounce buffer allocations.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Acked-by: NChristian König <christian.koenig@amd.com>

To properly reject too small DMA masks based on the addressability of the
bounce buffer.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Acked-by: NChristian König <christian.koenig@amd.com>

Currently all architectures that want to use swiotlb have to implement
their own dma_map_ops instances.  Provide a generic one based on the
x86 implementation which first calls into dma_direct to try a full blown
direct mapping implementation (including e.g. CMA) before falling back
allocating from the swiotlb buffer.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Acked-by: NChristian König <christian.koenig@amd.com>

Signed-off-by: NChristoph Hellwig <hch@lst.de>
Acked-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NKonrad Rzeszutek Wilk <konrad.wilk@oracle.com>

We'll need that name for a generic implementation soon.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Acked-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NKonrad Rzeszutek Wilk <konrad.wilk@oracle.com>

We'll need that name for a generic implementation soon.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Acked-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NKonrad Rzeszutek Wilk <konrad.wilk@oracle.com>

We'll need that name for a generic implementation soon.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Acked-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NKonrad Rzeszutek Wilk <konrad.wilk@oracle.com>

We'll need that name for a generic implementation soon.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Acked-by: NChristian König <christian.koenig@amd.com>
Reviewed-by: NRobin Murphy <robin.murphy@arm.com>

TTM tries to allocate coherent memory in chunks of 2MB first to improve
TLB efficiency and falls back to allocating 4K pages if that fails.

Suppress the warning when the 2MB allocations fails since there is a
valid fall back path.
Signed-off-by: NChristian König <christian.koenig@amd.com>
Reported-by: NMike Galbraith <efault@gmx.de>
Acked-by: NKonrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Bug: https://bugs.freedesktop.org/show_bug.cgi?id=104082
CC: stable@vger.kernel.org
Signed-off-by: NChristoph Hellwig <hch@lst.de>

Replace the bare-bones h8300 direct dma mapping implementation with
the fully featured generic dma-direct one.
Signed-off-by: NChristoph Hellwig <hch@lst.de>

cris currently has an incomplete direct mapping dma_map_ops implementation
if PCI support is enabled.  Replace it with the fully feature generic
dma-direct implementation.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Acked-by: NJesper Nilsson <jesper.nilsson@axis.com>

Signed-off-by: NChristoph Hellwig <hch@lst.de>
Reviewed-by: NRobin Murphy <robin.murphy@arm.com>

So that they don't need to indirect through the operation vector.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Reviewed-by: NVladimir Murzin <vladimir.murzin@arm.com>

If an attempt to allocate memory succeeded, but isn't inside the
supported DMA mask, retry the allocation with GFP_DMA set as a
last resort.

Based on the x86 code, but an off by one error in what is now
dma_coherent_ok has been fixed vs the x86 code.
Signed-off-by: NChristoph Hellwig <hch@lst.de>

This allows to dip into zones for lower memory if they are available.
If one of the zones is not available the corresponding GFP_* flag
will evaluate to 0 so they won't change anything.  We provide an
arch tunable for those architectures that do not use GFP_DMA for
the lowest 24-bits, given that there are a few.

Roughly based on the x86 code.
Signed-off-by: NChristoph Hellwig <hch@lst.de>

To preserve the x86 behavior.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Reviewed-by: NRobin Murphy <robin.murphy@arm.com>

Try the CMA allocator for coherent allocations if supported.

Roughly modelled after the x86 code.
Signed-off-by: NChristoph Hellwig <hch@lst.de>

Roughly based on the x86 pci-nommu implementation.
Signed-off-by: NChristoph Hellwig <hch@lst.de>

This means it uses whatever linear remapping scheme that the architecture
provides is used in the generic dma_direct ops.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Reviewed-by: NVladimir Murzin <vladimir.murzin@arm.com>

The trivial direct mapping implementation already does a virtual to
physical translation which isn't strictly a noop, and will soon learn
to do non-direct but linear physical to dma translations through the
device offset and a few small tricks.  Rename it to a better fitting
name.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Reviewed-by: NVladimir Murzin <vladimir.murzin@arm.com>

For architectures that just use the generic dma_noop_ops we can provide
a generic version of dma-mapping.h.
Signed-off-by: NChristoph Hellwig <hch@lst.de>

To implement the x86 forbid_dac and iommu_sac_force we want an arch hook
so that it can apply the global options across all dma_map_ops
implementations.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Reviewed-by: NKonrad Rzeszutek Wilk <konrad.wilk@oracle.com>

Lift the code from x86 so that we behave consistently.  In the future we
should probably warn if any of these is set.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Acked-by: NJesper Nilsson <jesper.nilsson@axis.com>
Acked-by: Geert Uytterhoeven <geert@linux-m68k.org> [m68k]

These days all devices should have a DMA coherent mask, and most dma_ops
implementations rely on that fact.  But just to be sure add an assert to
ring the warning bell if that is not the case.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Reviewed-by: NVladimir Murzin <vladimir.murzin@arm.com>
Reviewed-by: NKonrad Rzeszutek Wilk <konrad.wilk@oracle.com>

This is not needed in drivers, so move it to a private header.
Signed-off-by: NChristoph Hellwig <hch@lst.de>

Signed-off-by: NChristoph Hellwig <hch@lst.de>

Always returning 1 is the same behavior as not supplying a method at all.
Signed-off-by: NChristoph Hellwig <hch@lst.de>

This frees the dma_direct_* namespace for a generic implementation.
Signed-off-by: NChristoph Hellwig <hch@lst.de>

We want to use the dma_direct_ namespace for a generic implementation,
so rename powerpc to the second best choice: dma_nommu_.
Signed-off-by: NChristoph Hellwig <hch@lst.de>

Signed-off-by: NChristoph Hellwig <hch@lst.de>
Acked-by: NRichard Kuo <rkuo@codeaurora.org>

And unlike the other helpers we don't require a <asm/dma-direct.h> as
this helper is a special case for ia64 only, and this keeps it as
simple as possible.
Signed-off-by: NChristoph Hellwig <hch@lst.de>

phys_to_dma, dma_to_phys and dma_capable are helpers published by
architecture code for use of swiotlb and xen-swiotlb only.  Drivers are
not supposed to use these directly, but use the DMA API instead.

Move these to a new asm/dma-direct.h helper, included by a
linux/dma-direct.h wrapper that provides the default linear mapping
unless the architecture wants to override it.

In the MIPS case the existing dma-coherent.h is reused for now as
untangling it will take a bit of work.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Acked-by: NRobin Murphy <robin.murphy@arm.com>

This makes it match the generic version.
Reported-by: NVladimir Murzin <vladimir.murzin@arm.com>
Signed-off-by: NChristoph Hellwig <hch@lst.de>

The generic version now takes dma_pfn_offset into account, so there is no
more need for an architecture override.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Reviewed-by: NRobin Murphy <robin.murphy@arm.com>

This makes sure the generic version can be used with architectures /
devices that have a DMA offset in the direct mapping.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Reviewed-by: NRobin Murphy <robin.murphy@arm.com>

Signed-off-by: NChristoph Hellwig <hch@lst.de>

Signed-off-by: NChristoph Hellwig <hch@lst.de>

Signed-off-by: NChristoph Hellwig <hch@lst.de>