We only use ASIDs and VMIDs to identify individual stage-1 and stage-2
context-banks respectively, so rather than allocate these separately
from the context-banks, just calculate them based on the context bank
index.

Note that VMIDs are offset by 1, since VMID 0 is reserved for stage-1.
This doesn't cause us any issues with the numberspaces, since the
maximum number of context banks is half the minimum number of VMIDs.
Signed-off-by: NWill Deacon <will.deacon@arm.com>
Signed-off-by: NJoerg Roedel <joro@8bytes.org>

Although permitted by the architecture, using VMIDs for stage-1
translations causes a complete nightmare for hypervisors, who end up
having to virtualise the VMID space across VMs, which may be using
multiple VMIDs each.

To make life easier for hypervisors (which might just decide not to
support this VMID virtualisation), this patch reworks the stage-1
context-bank TLB invalidation so that:

	- Stage-1 mappings are marked non-global in the ptes
	- Each Stage-1 context-bank is assigned an ASID in TTBR0
	- VMID 0 is reserved for Stage-1 context-banks

This allows the hypervisor to overwrite the Stage-1 VMID in the CBAR
when trapping the write from the guest.
Signed-off-by: NWill Deacon <will.deacon@arm.com>
Signed-off-by: NJoerg Roedel <joro@8bytes.org>

On systems which use a single, combined irq line for the SMMU, context
faults may result in us spuriously reporting global faults with zero
status registers.

This patch fixes up the fsr checks in both the context and global fault
interrupt handlers, so that we only report the fault if the fsr
indicates something did indeed go awry.
Signed-off-by: NWill Deacon <will.deacon@arm.com>
Signed-off-by: NJoerg Roedel <joro@8bytes.org>

The bottom word of the pgd should always be written to the low half of
the TTBR, so we don't need to swap anything for big-endian.
Signed-off-by: NWill Deacon <will.deacon@arm.com>
Signed-off-by: NJoerg Roedel <joro@8bytes.org>

Fix two obvious problems:

1. We have registered msm_iommu_driver first, and need unregister it
   when registered msm_iommu_ctx_driver fail

2. We don't need to kfree drvdata before kzalloc was successful.

[akpm@linux-foundation.org: remove now-unneeded initialization of ctx_drvdata, remove unneeded braces]
Signed-off-by: NLibo Chen <libo.chen@huawei.com>
Acked-by: NDavid Brown <davidb@codeaurora.org>
Cc: David Woodhouse <dwmw2@infradead.org>
Cc: James Hogan <james.hogan@imgtec.com>
Signed-off-by: NAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>

This patch adds support for SMMUs implementing the ARM System MMU
architecture versions 1 or 2. Both arm and arm64 are supported, although
the v7s descriptor format is not used.

Cc: Rob Herring <robherring2@gmail.com>
Cc: Andreas Herrmann <andreas.herrmann@calxeda.com>
Cc: Olav Haugan <ohaugan@codeaurora.org>
Cc: Joerg Roedel <joro@8bytes.org>
Signed-off-by: NWill Deacon <will.deacon@arm.com>
Acked-by: NAndreas Herrmann <andreas.herrmann@calxeda.com>
Signed-off-by: NJoerg Roedel <joro@8bytes.org>

Calling clk_set_min_rate() is no better than just calling
clk_set_rate() because MSM clock code already takes care of
calling the min_rate ops if the clock really needs
clk_set_min_rate() called on it.

Cc: Joerg Roedel <joerg.roedel@amd.com>
Signed-off-by: NStephen Boyd <sboyd@codeaurora.org>
Acked-by: NJoerg Roedel <joro@8bytes.org>
Signed-off-by: NDavid Brown <davidb@codeaurora.org>

Add calls to clk_prepare and unprepare so that MSM can migrate to
the common clock framework. We never unprepare the clocks until
driver remove because the clocks are enabled and disabled in irq
context. Finer grained power management is possible in the future
via runtime power management techniques.

Cc: Joerg Roedel <joerg.roedel@amd.com>
Signed-off-by: NStephen Boyd <sboyd@codeaurora.org>
Acked-by: NJoerg Roedel <joro@8bytes.org>
Signed-off-by: NDavid Brown <davidb@codeaurora.org>

printk supports using %pa for phys_addr_t and
%zx for size_t so use those instead of %lx and
casts to unsigned long.

Other miscellaneous changes around this:

Always use 0x%zx for size instead of one use of decimal.
Coalesce format and align arguments.
Signed-off-by: NJoe Perches <joe@perches.com>
Signed-off-by: NJoerg Roedel <joro@8bytes.org>

If we use a large mapping, the expectation is that only unmaps from
the first pte in the superpage are supported.  Unmaps from offsets
into the superpage should fail (ie. return zero sized unmap).  In the
current code, unmapping from an offset clears the size of the full
mapping starting from an offset.  For instance, if we map a 16k
physically contiguous range at IOVA 0x0 with a large page, then
attempt to unmap 4k at offset 12k, 4 ptes are cleared (12k - 28k) and
the unmap returns 16k unmapped.  This potentially incorrectly clears
valid mappings and confuses drivers like VFIO that use the unmap size
to release pinned pages.

Fix by refusing to unmap from offsets into the page.
Signed-off-by: NAlex Williamson <alex.williamson@redhat.com>
Cc: stable@vger.kernel.org
Signed-off-by: NJoerg Roedel <joro@8bytes.org>

Signed-off-by: NAlex Williamson <alex.williamson@redhat.com>
Signed-off-by: NJoerg Roedel <joro@8bytes.org>

The IOMMU pagetables can have up to 6 levels, but the code
in free_pagetable() only releases the first 3 levels. Fix
this leak by releasing all levels.
Reported-by: NAlex Williamson <alex.williamson@redhat.com>
Signed-off-by: NJoerg Roedel <joro@8bytes.org>
Reviewed-by: NAlex Williamson <alex.williamson@redhat.com>

iommu_map splits requests into pages that the iommu driver reports
that it can handle.  The iommu_unmap path does not do the same.  This
can cause problems not only from callers that might expect the same
behavior as the map path, but even from the failure path of iommu_map,
should it fail at a point where it has mapped and needs to unwind a
set of pages that the iommu driver cannot handle directly.  amd_iommu,
for example, will BUG_ON if asked to unmap a non power of 2 size.

Fix this by extracting and generalizing the sizing code from the
iommu_map path and use it for both map and unmap.
Signed-off-by: NAlex Williamson <alex.williamson@redhat.com>
Signed-off-by: NJoerg Roedel <joro@8bytes.org>

If a device is multifunction and does not have ACS enabled then we
assume that the entire package lacks ACS and use function 0 as the
base of the group.  The PCIe spec however states that components are
permitted to implement ACS on some, none, or all of their applicable
functions.  It's therefore conceivable that function 0 may be fully
independent and support ACS while other functions do not.  Instead
use the lowest function of the slot that does not have ACS enabled
as the base of the group.  This may be the current device, which is
intentional.  So long as we use a consistent algorithm, all the
non-ACS functions will be grouped together and ACS functions will
get separate groups.
Signed-off-by: NAlex Williamson <alex.williamson@redhat.com>
Signed-off-by: NJoerg Roedel <joro@8bytes.org>

This patch fixes the checkpatch warnings in omap iommu
code, most of them are related to broken strings.
Signed-off-by: NSuman Anna <s-anna@ti.com>
Signed-off-by: NJoerg Roedel <joro@8bytes.org>

Fixed the following printk format warnings for dma_addr_t
for OMAP IOMMU.

drivers/iommu/omap-iommu.c: In function 'omap_iommu_iova_to_phys':
drivers/iommu/omap-iommu.c:1238:4: warning: format '%lx' expects type 'long unsigned int', but argument 4 has type 'dma_addr_t'
drivers/iommu/omap-iommu.c:1245:4: warning: format '%lx' expects type 'long unsigned int', but argument 4 has type 'dma_addr_t'
Signed-off-by: NSuman Anna <s-anna@ti.com>
Signed-off-by: NJoerg Roedel <joro@8bytes.org>

In intel vt-d spec , chapter 8.1 , DMA Remapping Reporting Structure.
In the end of the table, it says:

Remapping Structures[]
-
A list of structures. The list will contain one or
more DMA Remapping Hardware Unit Definition
(DRHD) structures, and zero or more Reserved
Memory Region Reporting (RMRR) and Root Port
ATS Capability Reporting (ATSR) structures.
These structures are described below.

So, there should be at least one DRHD structure in DMA Remapping
reporting table. If there is no DRHD found, a warning is necessary.
Signed-off-by: NLi, Zhen-Hua <zhen-hual@hp.com>
Signed-off-by: NJoerg Roedel <joro@8bytes.org>

This triggers on a MacBook Pro.
See https://bugzilla.redhat.com/show_bug.cgi?id=948262 for
the problem report.
Signed-off-by: NAndy Lutomirski <luto@amacapital.net>
Signed-off-by: NJoerg Roedel <joro@8bytes.org>

The enables VFIO on the pSeries platform, enabling user space
programs to access PCI devices directly.
Signed-off-by: NAlexey Kardashevskiy <aik@ozlabs.ru>
Cc: David Gibson <david@gibson.dropbear.id.au>
Signed-off-by: NPaul Mackerras <paulus@samba.org>
Acked-by: NAlex Williamson <alex.williamson@redhat.com>
Signed-off-by: NBenjamin Herrenschmidt <benh@kernel.crashing.org>

This initializes IOMMU groups based on the IOMMU configuration
discovered during the PCI scan on POWERNV (POWER non virtualized)
platform.  The IOMMU groups are to be used later by the VFIO driver,
which is used for PCI pass through.

It also implements an API for mapping/unmapping pages for
guest PCI drivers and providing DMA window properties.
This API is going to be used later by QEMU-VFIO to handle
h_put_tce hypercalls from the KVM guest.

The iommu_put_tce_user_mode() does only a single page mapping
as an API for adding many mappings at once is going to be
added later.

Although this driver has been tested only on the POWERNV
platform, it should work on any platform which supports
TCE tables.  As h_put_tce hypercall is received by the host
kernel and processed by the QEMU (what involves calling
the host kernel again), performance is not the best -
circa 220MB/s on 10Gb ethernet network.

To enable VFIO on POWER, enable SPAPR_TCE_IOMMU config
option and configure VFIO as required.

Cc: David Gibson <david@gibson.dropbear.id.au>
Signed-off-by: NAlexey Kardashevskiy <aik@ozlabs.ru>
Signed-off-by: NPaul Mackerras <paulus@samba.org>
Signed-off-by: NBenjamin Herrenschmidt <benh@kernel.crashing.org>

Add functionality to check the availability of the AMD IOMMU Performance
Counters and export this functionality to other core drivers, such as in this
case, a perf AMD IOMMU PMU.  This feature is not bound to any specific AMD
family/model other than the presence of the IOMMU with P-C enabled.

The AMD IOMMU P-C support static counting only at this time.
Signed-off-by: NSteven Kinney <steven.kinney@amd.com>
Signed-off-by: NSuravee Suthikulpanit <suravee.suthikulpanit@amd.com>
Signed-off-by: NPeter Zijlstra <peterz@infradead.org>
Link: http://lkml.kernel.org/r/1370466709-3212-2-git-send-email-suravee.suthikulpanit@amd.comSigned-off-by: NIngo Molnar <mingo@kernel.org>

Current multiple-MSI implementation does not take into account actual
number of requested MSIs and always rounds that number to a larger
power-of-two value.  Yet, the number of MSIs a PCI device could send (and
therefore the number of messages a device driver could request) may be
smaller.  As result, resources allocated for extra MSIs are just wasted.

This update takes advantage of 'msi_desc::nvec_used' field introduced with
generic MSI code to track the number of requested and used MSIs.  As
result, resources associated with interrupts are conserved.  Of those
resources most noticeable are x86 interrupt vectors.

The initial version of this fix also conserved IRTEs, but Jan noticed that
a malfunctioning PCI device might send a message number it did not claim
and thus refer to an IRTE it does not own.  To avoid this security hole,
as many IRTEs are reserved as the device could possibly send.

[bhelgaas: changelog, rename to "nvec_used"]
Signed-off-by: NAlexander Gordeev <agordeev@redhat.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Acked-by: NJoerg Roedel <joro@8bytes.org>

Current multiple-MSI implementation does not take into account actual
number of requested MSIs and always rounds that number to a larger
power-of-two value.  Yet, the number of MSIs a PCI device could send (and
therefore the number of messages a device driver could request) may be
smaller.  As result, resources allocated for extra MSIs are just wasted.

This update takes advantage of 'msi_desc::nvec_used' field introduced with
generic MSI code to track the number of requested and used MSIs.  As
result, resources associated with interrupts are conserved.  Of those
resources most noticeable are x86 interrupt vectors.

The initial version of this fix also conserved IRTEs, but Jan noticed that
a malfunctioning PCI device might send a message number it did not claim
and thus refer to an IRTE it does not own.  To avoid this security hole,
as many IRTEs are reserved as the device could possibly send.

[bhelgaas: changelog, rename to "nvec_used"]
Signed-off-by: NAlexander Gordeev <agordeev@redhat.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

Fix printk formats for dma_addr_t:

   drivers/iommu/tegra-smmu.c: In function 'smmu_iommu_iova_to_phys':
>> drivers/iommu/tegra-smmu.c:774:2: warning: format '%lx' expects argument of type 'long unsigned int', but argument 4 has type 'dma_addr_t' [-Wformat]
--
   drivers/iommu/tegra-gart.c: In function 'gart_iommu_iova_to_phys':
>> drivers/iommu/tegra-gart.c:298:3: warning: format '%lx' expects argument of type 'long unsigned int', but argument 3 has type 'dma_addr_t' [-Wformat]
Signed-off-by: NVarun Sethi <Varun.Sethi@freescale.com>
Signed-off-by: NJoerg Roedel <joro@8bytes.org>

As IOMMU groups are exposed to the user space by their numbers,
the user space can use them in various kernel APIs so the kernel
might need an API to find a group by its ID.

As an example, QEMU VFIO on PPC64 platform needs it to associate
a logical bus number (LIOBN) with a specific IOMMU group in order
to support in-kernel handling of DMA map/unmap requests.

The patch adds the iommu_group_get_by_id(id) function which performs
such search.

v2: fixed reference counting.
Signed-off-by: NAlexey Kardashevskiy <aik@ozlabs.ru>
Acked-by: NAlex Williamson <alex.williamson@redhat.com>
Signed-off-by: NJoerg Roedel <joro@8bytes.org>

ACPI_DMAR_SCOPE_TYPE_HPET is parsed by ir_parse_ioapic_hpet_scope() and
should not be flagged as an unsupported type.
Signed-off-by: NLinn Crosetto <linn@hp.com>
Signed-off-by: NJoerg Roedel <joro@8bytes.org>

ACPI_DMAR_SCOPE_TYPE_HPET is parsed by
ir_parse_ioapic_hpet_scope() and should not be flagged as an
unsupported type.
Signed-off-by: NLinn Crosetto <linn@hp.com>
Cc: joro@8bytes.org
Cc: ddutile@redhat.com
Cc: suresh.b.siddha@intel.com
Link: http://lkml.kernel.org/r/1366741605-71293-1-git-send-email-linn@hp.comSigned-off-by: NIngo Molnar <mingo@kernel.org>

The swap_pci_ref function is used by the IOMMU API code for
swapping pci device pointers, while determining the iommu
group for the device.
Currently this function was being implemented for different
IOMMU drivers.  This patch moves the function to a new file,
drivers/iommu/pci.h so that the implementation can be
shared across various IOMMU drivers.
Signed-off-by: NVarun Sethi <Varun.Sethi@freescale.com>
Signed-off-by: NJoerg Roedel <joro@8bytes.org>

This patch disables translation(dma-remapping) before its initialization
if it is already enabled.

This is needed for kexec/kdump boot. If dma-remapping is enabled in the
first kernel, it need to be disabled before initializing its page table
during second kernel boot. Wei Hu also reported that this is needed
when second kernel boots with intel_iommu=off.

Basically iommu->gcmd is used to know whether translation is enabled or
disabled, but it is always zero at boot time even when translation is
enabled since iommu->gcmd is initialized without considering such a
case. Therefor this patch synchronizes iommu->gcmd value with global
command register when iommu structure is allocated.
Signed-off-by: NTakao Indoh <indou.takao@jp.fujitsu.com>
Signed-off-by: NJoerg Roedel <joro@8bytes.org>

Fix to return -ENOMEM int the memory alloc error handling
case instead of 0, as done elsewhere in this function.
Signed-off-by: NWei Yongjun <yongjun_wei@trendmicro.com.cn>
Signed-off-by: NJoerg Roedel <joro@8bytes.org>

In the current interrupt handling scheme, there are as many threads as
the number of IOMMUs. Each thread is created and assigned to an IOMMU at
the time of registering interrupt handlers (request_threaded_irq).
When an IOMMU HW generates an interrupt, the irq handler (top half) wakes up
the corresponding thread to process event and PPR logs of all IOMMUs
starting from the 1st IOMMU.

In the system with multiple IOMMU,this handling scheme complicates the
synchronization of the IOMMU data structures and status registers as
there could be multiple threads competing for the same IOMMU while
the other IOMMU could be left unhandled.

To simplify, this patch is proposing a different interrupt handling scheme
by having each thread only managing interrupts of the corresponding IOMMU.
This can be achieved by passing the struct amd_iommu when registering the
interrupt handlers. This structure is unique for each IOMMU and can be used
by the bottom half thread to identify the IOMMU to be handled instead
of calling for_each_iommu.  Besides this also eliminate the needs to lock
the IOMMU for processing event and PPR logs.
Signed-off-by: NSuravee Suthikulpanit <suravee.suthikulpanit@amd.com>
Signed-off-by: NJoerg Roedel <joro@8bytes.org>

Work around an IOMMU  hardware bug where clearing the
EVT_INT or PPR_INT bit in the status register may race with
the hardware trying to set it again. When not handled the
bit might not be cleared and we lose all future event or ppr
interrupts.
Reported-by: NSuravee Suthikulpanit <suravee.suthikulpanit@amd.com>
Cc: stable@vger.kernel.org
Signed-off-by: NJoerg Roedel <joro@8bytes.org>

When the IVRS entries for IOAPIC and HPET are overridden on
the kernel command line, a problem detected in the check
function might not be a firmware bug anymore. So disable
the firmware bug reporting if the user provided valid
ivrs_ioapic or ivrs_hpet entries on the command line.
Reviewed-by: NShuah Khan <shuahkhan@gmail.com>
Signed-off-by: NJoerg Roedel <joro@8bytes.org>

Add two new kernel commandline parameters ivrs_ioapic and
ivrs_hpet to override the Id->DeviceId mapping from the IVRS
ACPI table. This can be used to work around broken BIOSes to
get interrupt remapping working on AMD systems.
Tested-by: NBorislav Petkov <bp@suse.de>
Tested-by: NSuravee Suthikulanit <suravee.suthikulpanit@amd.com>
Reviewed-by: NShuah Khan <shuahkhan@gmail.com>
Signed-off-by: NJoerg Roedel <joro@8bytes.org>

This is needed in a later patch were ioapic_map and hpet_map
entries are created before the slab allocator is initialized
(and thus add_special_device() can't be used).
Reviewed-by: NShuah Khan <shuahkhan@gmail.com>
Signed-off-by: NJoerg Roedel <joro@8bytes.org>

This patch extends the devid_map data structure to allow
ioapic and hpet entries in ivrs to be overridden on the
kernel command line.
Reviewed-by: NShuah Khan <shuahkhan@gmail.com>
Signed-off-by: NJoerg Roedel <joro@8bytes.org>

The function is only called by other __init functions, so it
can be moved to __init too.
Reviewed-by: NShuah Khan <shuahkhan@gmail.com>
Signed-off-by: NJoerg Roedel <joro@8bytes.org>

Fixes a lockdep warning.

Cc: stable@vger.kernel.org # >= v3.7
Reviewed-by: NShuah Khan <shuahkhan@gmail.com>
Signed-off-by: NJoerg Roedel <joro@8bytes.org>

For compatibility reasons the irq remapping code for the AMD
IOMMU used the same per-irq data structure as the Intel
implementation. Now that support for the AMD specific data
structure is upstream we can use this one instead.
Reviewed-by: NShuah Khan <shuahkhan@gmail.com>
Signed-off-by: NJoerg Roedel <joro@8bytes.org>

This function was intended as a fall-back if the map_sg
function is called for a device not mapped by the IOMMU.
Since the AMD IOMMU driver uses per-device dma_ops this can
never happen. So this function isn't needed anymore.
Reviewed-by: NShuah Khan <shuahkhan@gmail.com>
Signed-off-by: NJoerg Roedel <joro@8bytes.org>