Now that we can accurately reflect the context format we choose for each
domain, do that instead of imposing the global lowest-common-denominator
restriction and potentially ending up with nothing. We currently have a
strict 1:1 correspondence between domains and context banks, so we don't
need to entertain the possibility of multiple formats _within_ a domain.
Signed-off-by: NWill Deacon <will.deacon@arm.com>
[rm: split from original patch, added SMMUv3]
Signed-off-by: NRobin Murphy <robin.murphy@arm.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Now that we know exactly which page sizes our caller wants to use in the
given domain, we can restrict higher-order allocation attempts to just
those sizes, if any, and avoid wasting any time or effort on other sizes
which offer no benefit. In the same vein, this also lets us accommodate
a minimum order greater than 0 for special cases.
Signed-off-by: NRobin Murphy <robin.murphy@arm.com>
Acked-by: NWill Deacon <will.deacon@arm.com>
Tested-by: NYong Wu <yong.wu@mediatek.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Many IOMMUs support multiple page table formats, meaning that any given
domain may only support a subset of the hardware page sizes presented in
iommu_ops->pgsize_bitmap. There are also certain use-cases where the
creator of a domain may want to control which page sizes are used, for
example to force the use of hugepage mappings to reduce pagetable walk
depth.

To this end, add a per-domain pgsize_bitmap to represent the subset of
page sizes actually in use, to make it possible for domains with
different requirements to coexist.
Signed-off-by: NWill Deacon <will.deacon@arm.com>
[rm: hijacked and rebased original patch with new commit message]
Signed-off-by: NRobin Murphy <robin.murphy@arm.com>
Acked-by: NWill Deacon <will.deacon@arm.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

As a set of driver-provided callbacks and static data, there is no
compelling reason for struct iommu_ops to be mutable in core code, so
enforce const-ness throughout.
Acked-by: NThierry Reding <treding@nvidia.com>
Signed-off-by: NRobin Murphy <robin.murphy@arm.com>
Acked-by: NWill Deacon <will.deacon@arm.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Stop wasting IOVA space by over-aligning scatterlist segments for a
theoretical worst-case segment boundary mask, and instead take the real
limits into account to merge consecutive segments wherever appropriate,
so our callers can benefit from getting back nicely simplified lists.

This also represents the last piece of functionality wanted by users of
the current arch/arm implementation, thus brings us a small step closer
to converting that over to the common code.
Signed-off-by: NRobin Murphy <robin.murphy@arm.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

According MMU-500r2 TRM, section 3.7.1 Auxiliary Control registers,
You can modify ACTLR only when the ACR.CACHE_LOCK bit is 0.

So before clearing ARM_MMU500_ACTLR_CPRE of each context bank,
need clear CACHE_LOCK bit of ACR register first.

Since CACHE_LOCK bit is only present in MMU-500r2 onwards,
need to check the major number of IDR7.
Reviewed-by: NRobin Murphy <robin.murphy@arm.com>
Signed-off-by: NPeng Fan <van.freenix@gmail.com>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

The 64KB Translation Granule Supplement to the SMMUv1 architecture
allows an SMMUv1 implementation to support 64KB pages for stage 2
translations, using a constrained VMSAv8 descriptor format limited
to 40-bit addresses. Now that we can freely mix and match context
formats, we can actually handle having 4KB pages via an AArch32
context but 64KB pages via an AArch64 context, so plumb it in.

It is assumed that any implementations will have hardware capabilities
matching the format constraints, thus obviating the need for excessive
sanity-checking; this is the case for MMU-401, the only ARM Ltd.
implementation.

CC: Eric Auger <eric.auger@linaro.org>
Signed-off-by: NRobin Murphy <robin.murphy@arm.com>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

The way the driver currently forces an AArch32 or AArch64 context format
based on the kernel config and SMMU architecture version is suboptimal,
in that it makes it very hard to support oddball mix-and-match cases
like the SMMUv1 64KB supplement, or situations where the reduced table
depth of an AArch32 short descriptor context may be desirable under an
AArch64 kernel. It also only happens to work on current implementations
which do support all the relevant formats.

Introduce an explicit notion of context format, so we can manage that
independently and get rid of the inflexible #ifdeffery.
Signed-off-by: NRobin Murphy <robin.murphy@arm.com>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

With {read,write}q_relaxed now able to fall back to the common
nonatomic-hi-lo helper, make use of that so that we don't have to
open-code our own. In the process, also convert the other remaining
split accesses, and repurpose the custom accessor to smooth out the
couple of troublesome instances where we really want to avoid
nonatomic writes (and a 64-bit access is unnecessary in the 32-bit
context formats we would use on a 32-bit CPU).

This paves the way for getting rid of some of the assumptions currently
baked into the driver which make it really awkward to use 32-bit context
formats with SMMUv2 under a 64-bit kernel.
Signed-off-by: NRobin Murphy <robin.murphy@arm.com>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

MMU-500 erratum #841119 is tickled by a particular set of circumstances
interacting with the next-page prefetcher. Since said prefetcher is
quite dumb and actually detrimental to performance in some cases (by
causing unwanted TLB evictions for non-sequential access patterns), we
lose very little by turning it off, and what we gain is a guarantee that
the erratum is never hit.

As a bonus, the same workaround will also prevent erratum #826419 once
v7 short descriptor support is implemented.

CC: Catalin Marinas <catalin.marinas@arm.com>
CC: Will Deacon <will.deacon@arm.com>
Signed-off-by: NRobin Murphy <robin.murphy@arm.com>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

With a framework for implementation-specific funtionality in place, the
currently-FDT-dependent ThunderX workaround gets to be the first user.
Acked-by: NTirumalesh Chalamarla <tchalamarla@caviumnetworks.com>
Signed-off-by: NRobin Murphy <robin.murphy@arm.com>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

As the inevitable reality of implementation-specific errata workarounds
begin to accrue alongside our integration quirk handling, it's about
time the driver had a decent way of keeping track. Extend the per-SMMU
data so we can identify specific implementations in an efficient and
firmware-agnostic manner.
Acked-by: NTirumalesh Chalamarla <tchalamarla@caviumnetworks.com>
Signed-off-by: NRobin Murphy <robin.murphy@arm.com>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

Due to erratum #27704, the CN88xx SMMUv2 implementation supports only
shared ASID and VMID numberspaces.

This patch ensures that ASID and VMIDs are unique across all SMMU
instances on affected Cavium systems.
Signed-off-by: NTirumalesh Chalamarla <tchalamarla@caviumnetworks.com>
Signed-off-by: NAkula Geethasowjanya <Geethasowjanya.Akula@caviumnetworks.com>
[will: commit message, comments and formatting]
Signed-off-by: NWill Deacon <will.deacon@arm.com>

This patch adds support for 16-bit VMIDs on implementations of SMMUv2
that support it.
Signed-off-by: NTirumalesh Chalamarla <tchalamarla@caviumnetworks.com>
[will: commit messsage and comments]
Signed-off-by: NWill Deacon <will.deacon@arm.com>

Until we get fully plumbed into of_iommu_configure, our default
IOMMU_DOMAIN_DMA domains just bypass translation. Since we achieve that
by leaving the stream table entries set to bypass instead of pointing at
a translation context, the context bank we allocate for the domain is
completely wasted. Context banks are typically a rather limited
resource, so don't hog ones we don't need.
Reported-by: NEric Auger <eric.auger@linaro.org>
Tested-by: NEric Auger <eric.auger@linaro.org>
Signed-off-by: NRobin Murphy <robin.murphy@arm.com>
Signed-off-by: NWill Deacon <will.deacon@arm.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Commit cbf8277e ("iommu/arm-smmu: Treat IOMMU_DOMAIN_DMA as bypass
for now") ignores requests to attach a device to the default domain
since, without IOMMU-basked DMA ops available everywhere, the default
domain will just lead to unexpected transaction faults being reported.

Unfortunately, the way this was implemented on SMMUv2 causes a
regression with VFIO PCI device passthrough under KVM on AMD Seattle.
On this system, the host controller device is associated with both a
pci_dev *and* a platform_device, and can therefore end up with duplicate
SMR entries, resulting in a stream-match conflict at runtime.

This patch amends the original fix so that attaching to IOMMU_DOMAIN_DMA
is rejected even before configuring the SMRs. This restores the old
behaviour for now, but we'll need to look at handing host controllers
specially when we come to supporting the default domain fully.
Reported-by: NEric Auger <eric.auger@linaro.org>
Tested-by: NEric Auger <eric.auger@linaro.org>
Tested-by: NYang Shi <yang.shi@linaro.org>
Signed-off-by: NWill Deacon <will.deacon@arm.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Commit 61289cba ('iommu/amd: Remove old alias handling code')
removed the old alias handling code from the AMD IOMMU
driver because this is now handled by the IOMMU core code.

But this also removed the handling of PCI aliases, which is
not handled by the core code. This caused issues with PCI
devices that have hidden PCIe-to-PCI bridges that rewrite
the request-id.

Fix this bug by re-introducing some of the removed functions
from commit 61289cba and add a alias field
'struct iommu_dev_data'. This field carrys the return value
of the get_alias() function and uses that instead of the
amd_iommu_alias_table[] array in the code.

Fixes: 61289cba ('iommu/amd: Remove old alias handling code')
Cc: stable@vger.kernel.org # v4.4+
Tested-by: NTomasz Golinski <tomaszg@math.uwb.edu.pl>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

My static checker complains that "dma_alias" is uninitialized unless we
are dealing with a pci device.  This is true but harmless.  Anyway, we
can flip the condition around to silence the warning.
Signed-off-by: NDan Carpenter <dan.carpenter@oracle.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Since commit cd6438c5 ("iommu/rockchip: Reconstruct to support multi
slaves") rk_iommu_is_stall_active() always returns false because the
bitwise AND operates on the boolean flag promoted to an integer and a
value that is either zero or BIT(2).

Explicitly convert the right-hand value to a boolean so that both sides
are guaranteed to be either zero or one.

rk_iommu_is_paging_enabled() does not suffer from the same problem since
RK_MMU_STATUS_PAGING_ENABLED is BIT(0), but let's apply the same change
for consistency and to make it clear that it's correct without needing
to lookup the value.

Fixes: cd6438c5 ("iommu/rockchip: Reconstruct to support multi slaves")
Signed-off-by: NJohn Keeping <john@metanate.com>
Reviewed-by: NHeiko Stuebner <heiko@sntech.de>
Tested-by: NTomeu Vizoso <tomeu.vizoso@collabora.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

IOMMU drivers that do not support default domains, but make
use of the the group->domain pointer can get that pointer
overwritten with NULL on device add/remove.

Make sure this can't happen by only overwriting the domain
pointer when it is NULL.

Cc: stable@vger.kernel.org # v4.4+
Fixes: 1228236d ('iommu: Move default domain allocation to iommu_group_get_for_dev()')
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

With the change to stashing just the IOVA-page-aligned remainder of the
CPU-page offset rather than the whole thing, the failure path in
__invalidate_sg() also needs tweaking to account for that in the case of
differing page sizes where the two offsets may not be equivalent.
Similarly in __finalise_sg(), lest the architecture-specific wrappers
later get the wrong address for cache maintenance on sync or unmap.

Fixes: 164afb1d ("iommu/dma: Use correct offset in map_sg")
Reported-by: NMagnus Damm <damm+renesas@opensource.se>
Signed-off-by: NRobin Murphy <robin.murphy@arm.com>
Cc: stable@ver.kernel.org # v4.4+
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

of_platform_device_create() returns NULL on error, it never returns
error pointers.

Fixes: 0df4fabe ('iommu/mediatek: Add mt8173 IOMMU driver')
Signed-off-by: NDan Carpenter <dan.carpenter@oracle.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Whilst the default SLUB allocator happily just merges the original
allocation flags from kmem_cache_create() with those passed through
kmem_cache_alloc(), there is a code path in the SLAB allocator which
will aggressively BUG_ON() if the cache was created with SLAB_CACHE_DMA
but GFP_DMA is not specified for an allocation:

  kernel BUG at mm/slab.c:2536!
  Internal error: Oops - BUG: 0 [#1] SMP ARM
  Modules linked in:[    1.299311] Modules linked in:

  CPU: 1 PID: 1 Comm: swapper/0 Not tainted
  4.5.0-rc6-koelsch-05892-ge7e45ad53ab6795e #2270
  Hardware name: Generic R8A7791 (Flattened Device Tree)
  task: ef422040 ti: ef442000 task.ti: ef442000
  PC is at cache_alloc_refill+0x2a0/0x530
  LR is at _raw_spin_unlock+0x8/0xc
...
  [<c02c6928>] (cache_alloc_refill) from [<c02c6630>] (kmem_cache_alloc+0x7c/0xd4)
  [<c02c6630>] (kmem_cache_alloc) from [<c04444bc>]
  (__arm_v7s_alloc_table+0x5c/0x278)
  [<c04444bc>] (__arm_v7s_alloc_table) from [<c0444e1c>]
  (__arm_v7s_map.constprop.6+0x68/0x25c)
  [<c0444e1c>] (__arm_v7s_map.constprop.6) from [<c0445044>]
  (arm_v7s_map+0x34/0xa4)
  [<c0445044>] (arm_v7s_map) from [<c0c18ee4>] (arm_v7s_do_selftests+0x140/0x418)
  [<c0c18ee4>] (arm_v7s_do_selftests) from [<c0201760>]
  (do_one_initcall+0x100/0x1b4)
  [<c0201760>] (do_one_initcall) from [<c0c00d4c>]
  (kernel_init_freeable+0x120/0x1e8)
  [<c0c00d4c>] (kernel_init_freeable) from [<c067a364>] (kernel_init+0x8/0xec)
  [<c067a364>] (kernel_init) from [<c0206b68>] (ret_from_fork+0x14/0x2c)
  Code: 1a000003 e7f001f2 e3130001 0a000000 (e7f001f2)
  ---[ end trace 190f6f6b84352efd ]---

Keep the peace by adding GFP_DMA when allocating a table.
Reported-by: NGeert Uytterhoeven <geert@linux-m68k.org>
Acked-by: NWill Deacon <will.deacon@arm.com>
Signed-off-by: NRobin Murphy <robin.murphy@arm.com>
Tested-by: NGeert Uytterhoeven <geert+renesas@glider.be>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

The function can return negative value so it should be assigned to signed
variable. The patch changes also type of related i variable to make code
more compact and coherent.

The problem has been detected using patch
scripts/coccinelle/tests/unsigned_lesser_than_zero.cocci.
Signed-off-by: NAndrzej Hajda <a.hajda@samsung.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

In the PCI hotplug path of the Intel IOMMU driver, replace
the usage of the BUS_NOTIFY_DEL_DEVICE notifier, which is
executed before the driver is unbound from the device, with
BUS_NOTIFY_REMOVED_DEVICE, which runs after that.

This fixes a kernel BUG being triggered in the VT-d code
when the device driver tries to unmap DMA buffers and the
VT-d driver already destroyed all mappings.
Reported-by: NStefani Seibold <stefani@seibold.net>
Cc: stable@vger.kernel.org # v4.3+
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Detach the device that is about to be removed from its
domain (if it has one) to clear any related state like DTE
entry and device's ATS state.
Reported-by: NKelly Zytaruk <Kelly.Zytaruk@amd.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

IOMMU_DMA does indeed depend on scatterlists having a DMA length, but
the NEED_SG_DMA_LENGTH symbol should be selected, not depended upon.
Reported-by: NArnd Bergmann <arnd@arndb.de>
Signed-off-by: NRobin Murphy <robin.murphy@arm.com>
Acked-by: NArnd Bergmann <arnd@arndb.de>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

When CONFIG_PM is unset, we get a harmless warning for this driver:

drivers/iommu/mtk_iommu.c:665:12: error: 'mtk_iommu_suspend' defined but not used [-Werror=unused-function]
drivers/iommu/mtk_iommu.c:680:12: error: 'mtk_iommu_resume' defined but not used [-Werror=unused-function]

Marking the functions as __maybe_unused gits rid of the two functions
and lets the compiler silently drop the object code, while still
doing syntax checking on them for build-time verification.
Signed-off-by: NArnd Bergmann <arnd@arndb.de>
Fixes: 0df4fabe ("iommu/mediatek: Add mt8173 IOMMU driver")
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

The newly added Mediatek IOMMU driver uses the IOMMU_DMA infrastructure,
but unlike other such drivers, it does not select 'ARM_DMA_USE_IOMMU',
which is a prerequisite, leading to a link error:

warning: (MTK_IOMMU) selects IOMMU_DMA which has unmet direct dependencies (IOMMU_SUPPORT && NEED_SG_DMA_LENGTH)
drivers/iommu/built-in.o: In function `iommu_put_dma_cookie':
mtk_iommu.c:(.text+0x11fe): undefined reference to `put_iova_domain'
drivers/iommu/built-in.o: In function `iommu_dma_init_domain':
mtk_iommu.c:(.text+0x1316): undefined reference to `init_iova_domain'
drivers/iommu/built-in.o: In function `__iommu_dma_unmap':
mtk_iommu.c:(.text+0x1380): undefined reference to `find_iova'

This adds the same select that the other drivers have. On a related
note, I wonder if we should just always select ARM_DMA_USE_IOMMU
whenever any IOMMU driver is enabled. Are there any cases where
we would enable an IOMMU but not use it?
Signed-off-by: NArnd Bergmann <arnd@arndb.de>
Fixes: 0df4fabe ("iommu/mediatek: Add mt8173 IOMMU driver")
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Drivers should use generic readl/writel calls to access HW registers, so
replace all __raw_readl/writel with generic version.
Suggested-by: NArnd Bergmann <arnd@arndb.de>
Signed-off-by: NMarek Szyprowski <m.szyprowski@samsung.com>
Acked-by: NArnd Bergmann <arnd@arndb.de>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

The exynos iommu driver changed an incorrect cast from pointer
to 'unsigned int' to an equally incorrect cast to a 'phys_addr_t',
which results in an obvious compile-time error when phys_addr_t
is wider than pointers are:

drivers/iommu/exynos-iommu.c: In function 'alloc_lv2entry':
drivers/iommu/exynos-iommu.c:918:32: error: cast from pointer to integer of different size [-Werror=pointer-to-int-cast]

The code does not actually want the physical address (which would
involve using virt_to_phys()), but just checks the alignment,
so we can change it to use a cast to uintptr_t instead.
Signed-off-by: NArnd Bergmann <arnd@arndb.de>
Fixes: 740a01ee ("iommu/exynos: Add support for v5 SYSMMU")
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

The AMD Family 15h Models 30h-3Fh (Kaveri) BIOS and Kernel Developer's
Guide omitted part of the BIOS IOMMU L2 register setup specification.
Without this setup the IOMMU L2 does not fully respect write permissions
when handling an ATS translation request.

The IOMMU L2 will set PTE dirty bit when handling an ATS translation with
write permission request, even when PTE RW bit is clear. This may occur by
direct translation (which would cause a PPR) or by prefetch request from
the ATC.

This is observed in practice when the IOMMU L2 modifies a PTE which maps a
pagecache page. The ext4 filesystem driver BUGs when asked to writeback
these (non-modified) pages.

Enable ATS write permission check in the Kaveri IOMMU L2 if BIOS has not.
Signed-off-by: NJay Cornwall <jay@jcornwall.me>
Cc: <stable@vger.kernel.org> # v3.19+
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

The setup code for the performance counters in the AMD IOMMU driver
tests whether the counters can be written. It tests to setup a counter
for device 00:00.0, which fails on systems where this particular device
is not covered by the IOMMU.

Fix this by not relying on device 00:00.0 but only on the IOMMU being
present.

Cc: stable@vger.kernel.org
Signed-off-by: NSuravee Suthikulpanit <Suravee.Suthikulpanit@amd.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

This patch adds support for mediatek m4u (MultiMedia Memory Management
Unit).
Signed-off-by: NYong Wu <yong.wu@mediatek.com>
Reviewed-by: NRobin Murphy <robin.murphy@arm.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Make use of ARCH_RENESAS in place of ARCH_SHMOBILE.

This is part of an ongoing process to migrate from ARCH_SHMOBILE to
ARCH_RENESAS the motivation for which being that RENESAS seems to be a more
appropriate name than SHMOBILE for the majority of Renesas ARM based SoCs.
Signed-off-by: NSimon Horman <horms+renesas@verge.net.au>
Acked-by: NGeert Uytterhoeven <geert+renesas@glider.be>
Acked-by: NLaurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

IOMMU core calls attach_device callback without detaching device from
the previous domain. This patch adds support for such unballanced calls.
Signed-off-by: NMarek Szyprowski <m.szyprowski@samsung.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

This patch adds support for v5 of SYSMMU controller, found in Samsung
Exynos 5433 SoCs. The main difference of v5 is support for 36-bit physical
address space and some changes in register layout and core clocks hanging.
This patch also adds support for ARM64 architecture, which is used by
Exynos 5433 SoCs.
Signed-off-by: NMarek Szyprowski <m.szyprowski@samsung.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

SYSMMU on some SoCs reports bogus values in VERSION register. Force
hardware version to 1.0 for such controllers. This patch also moves reading
version register to driver's probe() function.
Signed-off-by: NMarek Szyprowski <m.szyprowski@samsung.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

This patch simplifies the code for handling of flpdcache invalidation.
Signed-off-by: NMarek Szyprowski <m.szyprowski@samsung.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

This patch rewrites sysmmu_init_config function to make it easier to read
and understand.
Signed-off-by: NMarek Szyprowski <m.szyprowski@samsung.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>