Add a nearly-complete ARMv7 short descriptor implementation, omitting
only a few legacy and CPU-centric aspects which shouldn't be necessary
for IOMMU API use anyway.
Reviewed-by: NYong Wu <yong.wu@mediatek.com>
Tested-by: NYong Wu <yong.wu@mediatek.com>
Signed-off-by: NYong Wu <yong.wu@mediatek.com>
Signed-off-by: NRobin Murphy <robin.murphy@arm.com>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

In below commit alias DTE is set when its peripheral is
setting DTE. However there's a code bug here to wrongly
set the alias DTE, correct it in this patch.

commit e25bfb56
Author: Joerg Roedel <jroedel@suse.de>
Date:   Tue Oct 20 17:33:38 2015 +0200

    iommu/amd: Set alias DTE in do_attach/do_detach
Signed-off-by: NBaoquan He <bhe@redhat.com>
Tested-by: NMark Hounschell <markh@compro.net>
Cc: stable@vger.kernel.org # v4.4
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Fix a simple typo when disabling IOTLB on PCI(e) devices.

Fixes: b16d0cb9 ("iommu/vt-d: Always enable PASID/PRI PCI capabilities before ATS")
Cc: stable@vger.kernel.org  # v4.4
Signed-off-by: NJeremy McNicoll <jmcnicol@redhat.com>
Reviewed-by: NAlex Williamson <alex.williamson@redhat.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Trying to build a kernel for ARC with both options CONFIG_COMPILE_TEST
and CONFIG_IOMMU_IO_PGTABLE_LPAE enabled (e.g. as a result of "make
allyesconfig") results in the following build failure:

 | CC drivers/iommu/io-pgtable-arm.o
 | linux/drivers/iommu/io-pgtable-arm.c: In
 | function ‘__arm_lpae_alloc_pages’:
 | linux/drivers/iommu/io-pgtable-arm.c:221:3:
 | error: implicit declaration of function ‘dma_map_single’
 | [-Werror=implicit-function-declaration]
 | dma = dma_map_single(dev, pages, size, DMA_TO_DEVICE);
 | ^
 | linux/drivers/iommu/io-pgtable-arm.c:221:42:
 | error: ‘DMA_TO_DEVICE’ undeclared (first use in this function)
 | dma = dma_map_single(dev, pages, size, DMA_TO_DEVICE);
 | ^

Since IOMMU_IO_PGTABLE_LPAE depends on DMA API, io-pgtable-arm.c should
include linux/dma-mapping.h. This fixes the reported failure.

Cc: Alexey Brodkin <abrodkin@synopsys.com>
Cc: Vineet Gupta <vgupta@synopsys.com>
Cc: Joerg Roedel <joro@8bytes.org>
Signed-off-by: NLada Trimasova <ltrimas@synopsys.com>
Signed-off-by: NWill Deacon <will.deacon@arm.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Only check for error when iommu->iommu_dev has been assigned
and only assign drhd->iommu when the function can't fail
anymore.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

This adds the proper check to alloc_iommu to make sure that
the call to iommu_device_create has completed successfully
and if not return the error code to the caller after freeing
up resources allocated previously.
Signed-off-by: NNicholas Krause <xerofoify@gmail.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

When mapping a non-page-aligned scatterlist entry, we copy the original
offset to the output DMA address before aligning it to hand off to
iommu_map_sg(), then later adding the IOVA page address portion to get
the final mapped address. However, when the IOVA page size is smaller
than the CPU page size, it is the offset within the IOVA page we want,
not that within the CPU page, which can easily be larger than an IOVA
page and thus result in an incorrect final address.

Fix the bug by taking only the IOVA-aligned part of the offset as the
basis of the DMA address, not the whole thing.
Signed-off-by: NRobin Murphy <robin.murphy@arm.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

get_device_id() returns an unsigned short device id.  It never fails and
it never returns a negative so we can remove this condition.
Signed-off-by: NDan Carpenter <dan.carpenter@oracle.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Preallocate between 4 and 8 apertures when a device gets it
dma_mask. With more apertures we reduce the lock contention
of the domain lock significantly.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

First search for a non-contended aperture with trylock
before spinning.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Make this pointer percpu so that we start searching for new
addresses in the range we last stopped and which is has a
higher probability of being still in the cache.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Remove the long holding times of the domain->lock and rely
on the bitmap_lock instead.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Make sure the aperture range is fully initialized before it
is visible to the address allocator.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

This allows to build up the page-tables without holding any
locks. As a consequence it removes the need to pre-populate
dma_ops page-tables.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

It really belongs there and not in __map_single.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Don't flush the iommu tlb when we free something behind the
current next_bit pointer. Update the next_bit pointer
instead and let the flush happen on the next wraparound in
the allocation path.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

The flushing of iommu tlbs is now done on a per-range basis.
So there is no need anymore for domain-wide flush tracking.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

This way we don't need to care about the next_index wrapping
around in dma_ops_alloc_addresses.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Instead of setting need_flush, do the flush directly in
dma_ops_free_addresses.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

It points to the next aperture index to allocate from. We
don't need the full address anymore because this is now
tracked in struct aperture_range.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Parameter is not needed because the value is part of the
already passed in struct dma_ops_domain.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Since the allocator wraparound happens in this function now,
flush the iommu tlb there too.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Instead of skipping to the next aperture, first try again in
the current one.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Moving it before the pte_pages array puts in into the same
cache-line as the spin-lock and the bitmap array pointer.
This should safe a cache-miss.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Make this a wrapper around iommu_ops_area_alloc() for now
and add more logic to this function later on.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

The page-offset of the aperture must be passed instead of 0.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

This allows to keep the bitmap_lock only for a very short
period of time.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

There have been present PTEs which in theory could have made
it to the IOMMU TLB. Flush the addresses out on the error
path to make sure no stale entries remain.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

This lock only protects the address allocation bitmap in one
aperture.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

It is only used in this file anyway, so keep it there. Same
with 'struct aperture_range'.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

This prevents possible flooding of the kernel log.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

If CONFIG_PHYS_ADDR_T_64BIT=n:

    drivers/iommu/ipmmu-vmsa.c: In function 'ipmmu_domain_init_context':
    drivers/iommu/ipmmu-vmsa.c:434:2: warning: right shift count >= width of type
      ipmmu_ctx_write(domain, IMTTUBR0, ttbr >> 32);
      ^

As io_pgtable_cfg.arm_lpae_s1_cfg.ttbr[] is an array of u64s, assigning
it to a phys_addr_t may truncates it.  Make ttbr u64 to fix this.
Signed-off-by: NGeert Uytterhoeven <geert+renesas@glider.be>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Doug reports that the equivalent page allocator on 32-bit ARM exhibits
particularly pathalogical behaviour under memory pressure when
fragmentation is high, where allocating a 4MB buffer takes tens of
seconds and the number of calls to alloc_pages() is over 9000![1]

We can drastically improve that situation without losing the other
benefits of high-order allocations when they would succeed, by assuming
memory pressure is relatively constant over the course of an allocation,
and not retrying allocations at orders we know to have failed before.
This way, the best-case behaviour remains unchanged, and in the worst
case we should see at most a dozen or so (MAX_ORDER - 1) failed attempts
before falling back to single pages for the remainder of the buffer.

[1]:http://lists.infradead.org/pipermail/linux-arm-kernel/2015-December/394660.htmlReported-by: NDouglas Anderson <dianders@chromium.org>
Signed-off-by: NRobin Murphy <robin.murphy@arm.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

dma-iommu.c was naughtily relying on an implicit transitive #include of
linux/vmalloc.h, which is apparently not present on some architectures.
Add that, plus a couple more headers for other functions which are used
similarly.
Reported-by: Nkbuild test robot <lkp@intel.com>
Signed-off-by: NRobin Murphy <robin.murphy@arm.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Those are stupid and code should use static_cpu_has_safe() or
boot_cpu_has() instead. Kill the least used and unused ones.

The remaining ones need more careful inspection before a conversion can
happen. On the TODO.
Signed-off-by: NBorislav Petkov <bp@suse.de>
Link: http://lkml.kernel.org/r/1449481182-27541-4-git-send-email-bp@alien8.de
Cc: David Sterba <dsterba@suse.com>
Cc: Herbert Xu <herbert@gondor.apana.org.au>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Matt Mackall <mpm@selenic.com>
Cc: Chris Mason <clm@fb.com>
Cc: Josef Bacik <jbacik@fb.com>
Signed-off-by: NThomas Gleixner <tglx@linutronix.de>

When tearing down page tables, we return early for the final level
since we know that we won't have any table pointers to follow.
Unfortunately, this also means that we forget to free the final level,
so we end up leaking memory.

Fix the issue by always freeing the current level, but just don't bother
to iterate over the ptes if we're at the final level.

Cc: <stable@vger.kernel.org>
Reported-by: NZhang Bo <zhangbo_a@xiaomi.com>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

It is ILLEGAL to set STE.S1STALLD to 1 if stage 1 is enabled and
either the stall or terminate models are not supported.

This patch fixes the STALLD check and ensures that we don't set STALLD
in the STE when it is not supported.
Signed-off-by: NPrem Mallappa <pmallapp@broadcom.com>
[will: consistently use IDR0_STALL_MODEL_* prefix]
Signed-off-by: NWill Deacon <will.deacon@arm.com>

When acknowledging global errors, the GERRORN register should be written
with the original GERROR value so that active errors are toggled.

This patch fixed the driver to write the original GERROR value to
GERRORN, instead of an active error mask.
Signed-off-by: NPrem Mallappa <pmallapp@broadcom.com>
[will: reworked use of active bits and fixed commit log]
Signed-off-by: NWill Deacon <will.deacon@arm.com>

There is no need to keep a useful accessor for a public structure hidden
away in a private implementation. Move it out alongside the structure
definition so that other implementations may reuse it.
Acked-by: NLaurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: NRobin Murphy <robin.murphy@arm.com>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

When invalidating an IOVA range potentially spanning multiple pages,
such as when removing an entire intermediate-level table, we currently
only issue an invalidation for the first IOVA of that range. Since the
architecture specifies that address-based TLB maintenance operations
target a single entry, an SMMU could feasibly retain live entries for
subsequent pages within that unmapped range, which is not good.

Make sure we hit every possible entry by iterating over the whole range
at the granularity provided by the pagetable implementation.
Signed-off-by: NRobin Murphy <robin.murphy@arm.com>
[will: added missing semicolons...]
Signed-off-by: NWill Deacon <will.deacon@arm.com>