Mapping and unmapping are more often than not in the critical path.
map_sg allows IOMMU driver implementations to optimize the process
of mapping buffers into the IOMMU page tables.

Instead of mapping a buffer one page at a time and requiring potentially
expensive TLB operations for each page, this function allows the driver
to map all pages in one go and defer TLB maintenance until after all
pages have been mapped.

Additionally, the mapping operation would be faster in general since
clients does not have to keep calling map API over and over again for
each physically contiguous chunk of memory that needs to be mapped to a
virtually contiguous region.
Signed-off-by: NOlav Haugan <ohaugan@codeaurora.org>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

The VT-d specification states that an RMRR entry in the DMAR
table needs to specify the full path to the device. This is
also how newer Linux kernels implement it.

Unfortunatly older drivers just match for the target device
and not the full path to the device, so that BIOS vendors
implement that behavior into their BIOSes to make them work
with older Linux kernels. But those RMRR entries break on
newer Linux kernels.

Work around this issue by adding a fall-back into the RMRR
matching code to match those old RMRR entries too.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

This will be used later to match broken RMRR entries.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

This makes sure any RMRR mappings stay in place when the
driver is unbound from the device.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>
Tested-by: NJerry Hoemann <jerry.hoemann@hp.com>

When the device id for an IOAPIC is overridden on the kernel
command line, the iommu driver has to make sure it sets up a
DTE for this device id.
Reported-by: NSu Friendy <friendy.su@sony.com.cn>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Commit 71054d88 ("x86, hpet: Introduce x86_msi_ops.setup_hpet_msi")
introduced x86_msi_ops.setup_hpet_msi to setup hpet MSI irq
when irq remapping enabled. This caused a regression of
hpet MSI irq remapping.

Original code flow before commit 71054d88:
hpet_setup_msi_irq()
	arch_setup_hpet_msi()
		setup_hpet_msi_remapped()
			remap_ops->setup_hpet_msi()
				alloc_irte()
		msi_compose_msg()
		hpet_msi_write()
		...

Current code flow after commit 71054d88:
hpet_setup_msi_irq()
	x86_msi.setup_hpet_msi()
		setup_hpet_msi_remapped()
			intel_setup_hpet_msi()
				alloc_irte()

Currently, we only call alloc_irte() for hpet MSI, but
do not composed and wrote its msg...
Signed-off-by: NYijing Wang <wangyijing@huawei.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

iommu_bus_init() registers a bus notifier on the given bus by using
a statically defined notifier block:

  static struct notifier_block iommu_bus_nb = {
          .notifier_call = iommu_bus_notifier,
  };

This same notifier block is used for all busses. This causes a
problem for notifiers registered after iommu has registered this
callback on multiple busses. The problem is that a subsequent
notifier being registered on a bus which has this iommu notifier
will also get linked in to the notifier list of all other busses
which have this iommu notifier.

This patch fixes this by allocating the notifier_block at runtime.
Some error checking is also added to catch any allocation failure
or notifier registration error.
Signed-off-by: NMark Salter <msalter@redhat.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

For a PCI device, aliases from the IVRS table won't be populated
into dma_alias_devfn until after iommu_init_device() is called on
each device.  We therefore want to split init_iommu_group() to
be called from a separate loop immediately following.
Signed-off-by: NAlex Williamson <alex.williamson@redhat.com>
Cc: stable@vger.kernel.org # 3.17
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

It turns out that our assumption that aliases are always to the same
slot isn't true.  One particular platform reports an IVRS alias of the
SATA controller (00:11.0) for the legacy IDE controller (00:14.1).
When we hit this, we attempt to use a single IOMMU group for
everything on the same bus, which in this case is the root complex.
We already have multiple groups defined for the root complex by this
point, resulting in multiple WARN_ON hits.

This patch makes these sorts of aliases work again with IOMMU groups
by reworking how we search through the PCI address space to find
existing groups.  This should also now handle looped dependencies and
all sorts of crazy inter-dependencies that we'll likely never see.

The recursion used here should never be very deep.  It's unlikely to
have individual aliases and only theoretical that we'd ever see a
chain where one alias causes us to search through to yet another
alias.  We're also only dealing with PCIe device on a single bus,
which means we'll typically only see multiple slots in use on the root
complex.  Loops are also a theoretically possibility, which I've
tested using fake DMA alias quirks and prevent from causing problems
using a bitmap of the devfn space that's been visited.
Signed-off-by: NAlex Williamson <alex.williamson@redhat.com>
Cc: stable@vger.kernel.org # 3.17
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Make of_device_id array const, because all OF functions handle it as const.
Signed-off-by: NKiran Padwal <kiran.padwal@smartplayin.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

under low memory conditions, alloc_pte() may return a NULL pointer.
iommu_map_page() does not check it and will panic the system.
Signed-off-by: NMaurizio Lombardi <mlombard@redhat.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

The owner field is never set. Remove it.
Signed-off-by: NLaurent Pinchart <laurent.pinchart@ideasonboard.com>
Acked-by: NSuman Anna <s-anna@ti.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Cc: Hiroshi Doyu <hdoyu@nvidia.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Cc: Jiang Liu <jiang.liu@linux.intel.com>
Cc: David Woodhouse <dwmw2@infradead.org>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Cc: Varun Sethi <Varun.Sethi@freescale.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Cc: Will Deacon <will.deacon@arm.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Signed-off-by: NJoerg Roedel <jroedel@suse.de>

This function will replace the current iommu_domain_has_cap
function and clean up the interface while at it.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Allow compile-time type-checking.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

1. We were calling clear_flush_young_notify in unmap_one, but we are
within an mmu notifier invalidate range scope. The spte exists no more
(due to range_start) and the accessed bit info has already been
propagated (due to kvm_pfn_set_accessed). Simply call
clear_flush_young.

2. We clear_flush_young on a primary MMU PMD, but this may be mapped
as a collection of PTEs by the secondary MMU (e.g. during log-dirty).
This required expanding the interface of the clear_flush_young mmu
notifier, so a lot of code has been trivially touched.

3. In the absence of shadow_accessed_mask (e.g. EPT A bit), we emulate
the access bit by blowing the spte. This requires proper synchronizing
with MMU notifier consumers, like every other removal of spte's does.
Signed-off-by: NAndres Lagar-Cavilla <andreslc@google.com>
Acked-by: NRik van Riel <riel@redhat.com>
Signed-off-by: NPaolo Bonzini <pbonzini@redhat.com>

We are using the same pfn for every pte we create while constructing the
pmd. Fix this by actually updating the pfn on each iteration of the pmd
construction loop.

It's not clear if we can actually hit this bug right now since iommu_map
splits up the calls to .map based on the page size, so we only ever seem to
iterate this loop once. However, things might change in the future that
might cause us to hit this.
Signed-off-by: NMitchel Humpherys <mitchelh@codeaurora.org>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

MMU-401 is similar to MMU-400, but updated with limited ARMv8 support.
Signed-off-by: NRobin Murphy <robin.murphy@arm.com>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

The SMMU driver was relying on a quirk of MMU-500 r2px to identify
the correct architecture version. Since this does not apply to other
implementations, make the architecture version for each supported
implementation explicit.

While we're at it, remove the unnecessary #ifdef since the dependencies
for CONFIG_ARM_SMMU already imply CONFIG_OF.
Signed-off-by: NRobin Murphy <robin.murphy@arm.com>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

In order for nested translation to work correctly, we need to ensure
that the maximum output address size from stage-1 is <= the maximum
supported input address size to stage-2. The latter is currently defined
by VA_BITS, since we make use of the CPU page table functions for
allocating out tables and so the driver currently enforces this
restriction by truncating the stage-1 output size during probe.

In reality, this doesn't make a lot of sense; the guest OS is responsible
for managing the stage-1 page tables, so we actually just need to ensure
that the ID registers of the virtual SMMU interface only advertise the
supported stage-2 input size.

This patch fixes the problem by treating the stage-1 and stage-2 input
address sizes separately.
Reported-by: NTirumalesh Chalamarla <tchalamarla@cavium.com>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

Arbitrary integer division is not available in all ARM CPUs, so the GCC
may spit out calls to helper functions which are not implemented in
the kernel.

This patch avoids these problems in the SMMU driver by using page shift
instead of page size, so that divisions by the page size (as required
by the vSMMU code) can be expressed as a simple right shift.
Signed-off-by: NWill Deacon <will.deacon@arm.com>

In preparation for nested translation support, stick a pointer to the
iommu_domain in dev->archdata.iommu. This makes it much easier to grab
hold of the physical group configuration (e.g. cbndx) when dealing with
vSMMU accesses from a guest.
Signed-off-by: NWill Deacon <will.deacon@arm.com>

Whilst the driver currently creates one IOMMU group per device, this
will soon change when we start supporting non-transparent PCI bridges
which require all upstream masters to be assigned to the same address
space.

This patch reworks our IOMMU group code so that we can easily support
multi-master groups. The master configuration (streamids and smrs) is
stored as private iommudata on the group, whilst the low-level attach/detach
code is updated to avoid double alloc/free when dealing with multiple
masters sharing the same SMMU configuration. This unifies device
handling, regardless of whether the device sits on the platform or pci
bus.
Signed-off-by: NWill Deacon <will.deacon@arm.com>

When debugging and testing code on an SMMU that supports nested
translation, it can be useful to restrict the driver to a particular
stage of translation.

This patch adds a module parameter to the ARM SMMU driver to allow this
by restricting the ability of the probe() code to detect support for
only the specified stage.
Signed-off-by: NWill Deacon <will.deacon@arm.com>

iommu_group_get_for_dev determines the iommu group for the PCI device and adds
the device to the group.

In the PAMU driver we were again adding the device to the same group without checking
if the device already had an iommu group. This resulted in the following warning.

sysfs: cannot create duplicate filename '/devices/ffe200000.pcie/pci0000:00/0000:00:00.0/iommu_group'
------------[ cut here ]------------
WARNING: at fs/sysfs/dir.c:31
Modules linked in:
CPU: 0 PID: 1 Comm: swapper/0 Not tainted 3.17.0-rc3-00002-g7505ceaf-dirty #126
task: c0000001fe0a0000 ti: c0000001fe044000 task.ti: c0000001fe044000
NIP: c00000000018879c LR: c000000000188798 CTR: c00000000001ea50
REGS: c0000001fe047040 TRAP: 0700   Not tainted  (3.17.0-rc3-00002-g7505ceaf-dirty)
MSR: 0000000080029000 <CE,EE,ME>  CR: 24ad8e22  XER: 20000000
SOFTE: 1
GPR00: c000000000188798 c0000001fe0472c0 c0000000009a52e0 0000000000000065
GPR04: 0000000000000001 0000000000000000 3a30303a00000000 0000000027000000
GPR08: 2f696f6d00000000 c0000000008d3830 c0000000009b3938 c0000000009bb3d0
GPR12: 0000000028ad8e24 c00000000fff4000 c00000000000205c 0000000000000000
GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
GPR20: 0000000000000000 0000000000000000 0000000000000000 c0000000008a4c70
GPR24: c0000000007e9010 c0000001fe0140a8 ffffffffffffffef 0000000000000001
GPR28: c0000001fe22ebb8 c0000000007e9010 c00000000090bf10 c0000001fe220000
NIP [c00000000018879c] .sysfs_warn_dup+0x74/0xa4
LR [c000000000188798] .sysfs_warn_dup+0x70/0xa4
Call Trace:
[c0000001fe0472c0] [c000000000188798] .sysfs_warn_dup+0x70/0xa4 (unreliable)
[c0000001fe047350] [c000000000188d34] .sysfs_do_create_link_sd.clone.2+0x168/0x174
[c0000001fe047400] [c0000000004b3cf8] .iommu_group_add_device+0x78/0x244
[c0000001fe0474b0] [c0000000004b6964] .fsl_pamu_add_device+0x88/0x1a8
[c0000001fe047570] [c0000000004b3960] .iommu_bus_notifier+0xdc/0x15c
[c0000001fe047600] [c000000000059848] .notifier_call_chain+0x8c/0xe8
[c0000001fe0476a0] [c000000000059d04] .__blocking_notifier_call_chain+0x58/0x84
[c0000001fe047750] [c00000000036619c] .device_add+0x464/0x5c8
[c0000001fe047820] [c000000000300ebc] .pci_device_add+0x14c/0x17c
[c0000001fe0478c0] [c000000000300fbc] .pci_scan_single_device+0xd0/0xf4
[c0000001fe047970] [c00000000030104c] .pci_scan_slot+0x6c/0x18c
[c0000001fe047a10] [c00000000030226c] .pci_scan_child_bus+0x40/0x114
[c0000001fe047ac0] [c000000000021974] .pcibios_scan_phb+0x240/0x2c8
[c0000001fe047b70] [c00000000085a970] .pcibios_init+0x64/0xc8
[c0000001fe047c00] [c000000000001884] .do_one_initcall+0xbc/0x224
[c0000001fe047d00] [c000000000852d50] .kernel_init_freeable+0x14c/0x21c
[c0000001fe047db0] [c000000000002078] .kernel_init+0x1c/0xfa4
[c0000001fe047e30] [c000000000000884] .ret_from_kernel_thread+0x58/0xd4
Instruction dump:
7c7f1b79 4182001c 7fe4fb78 7f83e378 38a01000 4bffc905 60000000 7c641b78
e87e8008 7fa5eb78 48482ff5 60000000 <0fe00000> 7fe3fb78 4bf7bd39 60000000
Signed-off-by: NVarun Sethi <Varun.Sethi@freescale.com>
Cc: stable@vger.kernel.org
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

A device is tied to an iommu through its archdata field. The archdata
is allocated on the fly for DT-based devices automatically through the
.add_device iommu ops. The current logic incorrectly assigned the name
of the IOMMU user device, instead of the name of the IOMMU device as
required by the attach logic. Fix this issue so that DT-based devices
can attach successfully to an IOMMU domain.
Signed-off-by: NSuman Anna <s-anna@ti.com>
Acked-by: NLaurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Any device requiring to be attached to an iommu_domain must have
valid archdata containing the necessary iommu information, which
is SoC-specific. Add a check in the omap_iommu_attach_dev to make
sure that the device has valid archdata before accessing
different SoC-specific fields of the archdata. This prevents a
NULL pointer dereference on any misconfigured devices.
Signed-off-by: NSuman Anna <s-anna@ti.com>
Acked-by: NLaurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Working out the usable address sizes for the SMMU is surprisingly tricky.
We must take into account both the limitations of the hardware for VA,
IPA and PA sizes but also any restrictions imposed by the Linux page
table code, particularly when dealing with nested translation (where the
IPA size is limited by the input address size at stage-2).

This patch fixes a few corner cases in our address size handling so that
we correctly deal with 40-bit addresses in TTBCR2 and restrict the IPA
size differently depending on whether or not we have support for nested
translation.
Signed-off-by: NWill Deacon <will.deacon@arm.com>

The prefix suggests the number should be printed in hex, so use
the %x specifier to do that.

Found by using regex suggested by Joe Perches.
Signed-off-by: NHans Wennborg <hans@hanshq.net>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

The number of S2CR registers is not properly set when stream
matching is not supported. Fix this and add check that we do not try to
access outside of the number of S2CR regisrers.
Signed-off-by: NOlav Haugan <ohaugan@codeaurora.org>
[will: added missing NUMSIDB_* definitions]
Signed-off-by: NWill Deacon <will.deacon@arm.com>

When we attach a device to a domain, we configure the SMRs (if we have
any) to match the Stream IDs for the corresponding SMMU master and
program the s2crs accordingly. However, on detach we tear down the s2crs
assuming stream-indexing (as opposed to stream-matching) and SMRs
assuming they are present.

This patch fixes the device detach code so that it operates as a
converse of the attach code.

Cc: <stable@vger.kernel.org>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

If split page table lock for PTE tables is enabled (CONFIG_SPLIT_PTLOCK_CPUS
<=NR_CPUS) pgtable_page_ctor() leads to non-atomic allocation for ptlock with
a spinlock held, resulting in:

------------[ cut here ]------------
WARNING: CPU: 0 PID: 466 at kernel/locking/lockdep.c:2742 lockdep_trace_alloc+0xd8/0xf4()
DEBUG_LOCKS_WARN_ON(irqs_disabled_flags(flags))
Modules linked in:
CPU: 0 PID: 466 Comm: dma0chan0-copy0 Not tainted 3.16.0-3d47efb-clean-pl330-dma_test-ve-a15-a32-slr-m
c-on-3+ #55
[<80014748>] (unwind_backtrace) from [<80011640>] (show_stack+0x10/0x14)
[<80011640>] (show_stack) from [<802bf864>] (dump_stack+0x80/0xb4)
[<802bf864>] (dump_stack) from [<8002385c>] (warn_slowpath_common+0x64/0x88)
[<8002385c>] (warn_slowpath_common) from [<80023914>] (warn_slowpath_fmt+0x30/0x40)
[<80023914>] (warn_slowpath_fmt) from [<8005d818>] (lockdep_trace_alloc+0xd8/0xf4)
[<8005d818>] (lockdep_trace_alloc) from [<800d3d78>] (kmem_cache_alloc+0x24/0x144)
[<800d3d78>] (kmem_cache_alloc) from [<800bfae4>] (ptlock_alloc+0x18/0x2c)
[<800bfae4>] (ptlock_alloc) from [<802b1ec0>] (arm_smmu_handle_mapping+0x4c0/0x690)
[<802b1ec0>] (arm_smmu_handle_mapping) from [<802b0cd8>] (iommu_map+0xe0/0x148)
[<802b0cd8>] (iommu_map) from [<80019098>] (arm_coherent_iommu_map_page+0x160/0x278)
[<80019098>] (arm_coherent_iommu_map_page) from [<801f4d78>] (dmatest_func+0x60c/0x1098)
[<801f4d78>] (dmatest_func) from [<8003f8ac>] (kthread+0xcc/0xe8)
[<8003f8ac>] (kthread) from [<8000e868>] (ret_from_fork+0x14/0x2c)
---[ end trace ce0d27e6f434acf8 ]--

Split page tables lock is not used in the driver. In fact, page tables are
guarded with domain lock, so remove calls to pgtable_page_{c,d}tor().

Cc: <stable@vger.kernel.org>
Signed-off-by: NVladimir Murzin <vladimir.murzin@arm.com>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

Stage-1 context banks do not have the SMMU_CBn_TCR[SL0] field since it
is only applicable to stage-2 context banks.

This patch ensures that we don't set the reserved TCR bits for stage-1
translations.

Cc: <stable@vger.kernel.org>
Signed-off-by: NOlav Haugan <ohaugan@codeaurora.org>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

request_irq shouldn't be called from atomic context since it might
sleep, but we're calling it with a spinlock held, resulting in:

    [    9.172202] BUG: sleeping function called from invalid context at kernel/mm/slub.c:926
    [    9.182989] in_atomic(): 1, irqs_disabled(): 128, pid: 1, name: swapper/0
    [    9.189762] CPU: 1 PID: 1 Comm: swapper/0 Tainted: G        W    3.10.40-gbc1b510b-38437-g55831d3bd9-dirty #97
    [    9.199757] [<c020c448>] (unwind_backtrace+0x0/0x11c) from [<c02097d0>] (show_stack+0x10/0x14)
    [    9.208346] [<c02097d0>] (show_stack+0x10/0x14) from [<c0301d74>] (kmem_cache_alloc_trace+0x3c/0x210)
    [    9.217543] [<c0301d74>] (kmem_cache_alloc_trace+0x3c/0x210) from [<c0276a48>] (request_threaded_irq+0x88/0x11c)
    [    9.227702] [<c0276a48>] (request_threaded_irq+0x88/0x11c) from [<c0931ca4>] (arm_smmu_attach_dev+0x188/0x858)
    [    9.237686] [<c0931ca4>] (arm_smmu_attach_dev+0x188/0x858) from [<c0212cd8>] (arm_iommu_attach_device+0x18/0xd0)
    [    9.247837] [<c0212cd8>] (arm_iommu_attach_device+0x18/0xd0) from [<c093314c>] (arm_smmu_test_probe+0x68/0xd4)
    [    9.257823] [<c093314c>] (arm_smmu_test_probe+0x68/0xd4) from [<c05aadd0>] (driver_probe_device+0x12c/0x330)
    [    9.267629] [<c05aadd0>] (driver_probe_device+0x12c/0x330) from [<c05ab080>] (__driver_attach+0x68/0x8c)
    [    9.277090] [<c05ab080>] (__driver_attach+0x68/0x8c) from [<c05a92d4>] (bus_for_each_dev+0x70/0x84)
    [    9.286118] [<c05a92d4>] (bus_for_each_dev+0x70/0x84) from [<c05aa3b0>] (bus_add_driver+0x100/0x244)
    [    9.295233] [<c05aa3b0>] (bus_add_driver+0x100/0x244) from [<c05ab5d0>] (driver_register+0x9c/0x124)
    [    9.304347] [<c05ab5d0>] (driver_register+0x9c/0x124) from [<c0933088>] (arm_smmu_test_init+0x14/0x38)
    [    9.313635] [<c0933088>] (arm_smmu_test_init+0x14/0x38) from [<c0200618>] (do_one_initcall+0xb8/0x160)
    [    9.322926] [<c0200618>] (do_one_initcall+0xb8/0x160) from [<c1200b7c>] (kernel_init_freeable+0x108/0x1cc)
    [    9.332564] [<c1200b7c>] (kernel_init_freeable+0x108/0x1cc) from [<c0b924b0>] (kernel_init+0xc/0xe4)
    [    9.341675] [<c0b924b0>] (kernel_init+0xc/0xe4) from [<c0205e38>] (ret_from_fork+0x14/0x3c)

Fix this by moving the request_irq out of the critical section. This
should be okay since smmu_domain->smmu is still being protected by the
critical section. Also, we still don't program the Stream Match Register
until after registering our interrupt handler so we shouldn't be missing
any interrupts.

Cc: <stable@vger.kernel.org>
Signed-off-by: NMitchel Humpherys <mitchelh@codeaurora.org>
[will: code cleanup and fixed request_irq token parameter]
Signed-off-by: NWill Deacon <will.deacon@arm.com>