This patch corrects the PASID format in the INVALIDATE_IOTLB_PAGES
command, which was caused by incorrect information in
the AMD IOMMU Architectural Specification v2.01 document.

    Incorrect format:
         cmd->data[0][16:23] = PASID[7:0]
         cmd->data[1][16:27] = PASID[19:8]

     Correct format:
         cmd->data[0][16:23] = PASID[15:8]
         cmd->data[1][16:23] = PASID[7:0]

However, this does not affect the IOMMUv2 hardware implementation,
and has been corrected since version 2.02 of the specification
(available through AMD NDA).
Signed-off-by: NJay Cornwall <jay.cornwall@amd.com>
Reviewed-by: NSuravee Suthikulpanit <suravee.suthikulpanit@amd.com>
Signed-off-by: NJoerg Roedel <joro@8bytes.org>

Use PCI standard marco dev_is_pci() instead of directly compare
pci_bus_type to check whether it is pci device.
Signed-off-by: NYijing Wang <wangyijing@huawei.com>
Signed-off-by: NJoerg Roedel <joro@8bytes.org>

Detected by cppcheck.
Signed-off-by: NKamil Dudka <kdudka@redhat.com>
Signed-off-by: NJoerg Roedel <joro@8bytes.org>

If we use a large mapping, the expectation is that only unmaps from
the first pte in the superpage are supported.  Unmaps from offsets
into the superpage should fail (ie. return zero sized unmap).  In the
current code, unmapping from an offset clears the size of the full
mapping starting from an offset.  For instance, if we map a 16k
physically contiguous range at IOVA 0x0 with a large page, then
attempt to unmap 4k at offset 12k, 4 ptes are cleared (12k - 28k) and
the unmap returns 16k unmapped.  This potentially incorrectly clears
valid mappings and confuses drivers like VFIO that use the unmap size
to release pinned pages.

Fix by refusing to unmap from offsets into the page.
Signed-off-by: NAlex Williamson <alex.williamson@redhat.com>
Cc: stable@vger.kernel.org
Signed-off-by: NJoerg Roedel <joro@8bytes.org>

The IOMMU pagetables can have up to 6 levels, but the code
in free_pagetable() only releases the first 3 levels. Fix
this leak by releasing all levels.
Reported-by: NAlex Williamson <alex.williamson@redhat.com>
Signed-off-by: NJoerg Roedel <joro@8bytes.org>
Reviewed-by: NAlex Williamson <alex.williamson@redhat.com>

If a device is multifunction and does not have ACS enabled then we
assume that the entire package lacks ACS and use function 0 as the
base of the group.  The PCIe spec however states that components are
permitted to implement ACS on some, none, or all of their applicable
functions.  It's therefore conceivable that function 0 may be fully
independent and support ACS while other functions do not.  Instead
use the lowest function of the slot that does not have ACS enabled
as the base of the group.  This may be the current device, which is
intentional.  So long as we use a consistent algorithm, all the
non-ACS functions will be grouped together and ACS functions will
get separate groups.
Signed-off-by: NAlex Williamson <alex.williamson@redhat.com>
Signed-off-by: NJoerg Roedel <joro@8bytes.org>

The swap_pci_ref function is used by the IOMMU API code for
swapping pci device pointers, while determining the iommu
group for the device.
Currently this function was being implemented for different
IOMMU drivers.  This patch moves the function to a new file,
drivers/iommu/pci.h so that the implementation can be
shared across various IOMMU drivers.
Signed-off-by: NVarun Sethi <Varun.Sethi@freescale.com>
Signed-off-by: NJoerg Roedel <joro@8bytes.org>

In the current interrupt handling scheme, there are as many threads as
the number of IOMMUs. Each thread is created and assigned to an IOMMU at
the time of registering interrupt handlers (request_threaded_irq).
When an IOMMU HW generates an interrupt, the irq handler (top half) wakes up
the corresponding thread to process event and PPR logs of all IOMMUs
starting from the 1st IOMMU.

In the system with multiple IOMMU,this handling scheme complicates the
synchronization of the IOMMU data structures and status registers as
there could be multiple threads competing for the same IOMMU while
the other IOMMU could be left unhandled.

To simplify, this patch is proposing a different interrupt handling scheme
by having each thread only managing interrupts of the corresponding IOMMU.
This can be achieved by passing the struct amd_iommu when registering the
interrupt handlers. This structure is unique for each IOMMU and can be used
by the bottom half thread to identify the IOMMU to be handled instead
of calling for_each_iommu.  Besides this also eliminate the needs to lock
the IOMMU for processing event and PPR logs.
Signed-off-by: NSuravee Suthikulpanit <suravee.suthikulpanit@amd.com>
Signed-off-by: NJoerg Roedel <joro@8bytes.org>

Work around an IOMMU  hardware bug where clearing the
EVT_INT or PPR_INT bit in the status register may race with
the hardware trying to set it again. When not handled the
bit might not be cleared and we lose all future event or ppr
interrupts.
Reported-by: NSuravee Suthikulpanit <suravee.suthikulpanit@amd.com>
Cc: stable@vger.kernel.org
Signed-off-by: NJoerg Roedel <joro@8bytes.org>

Fixes a lockdep warning.

Cc: stable@vger.kernel.org # >= v3.7
Reviewed-by: NShuah Khan <shuahkhan@gmail.com>
Signed-off-by: NJoerg Roedel <joro@8bytes.org>

For compatibility reasons the irq remapping code for the AMD
IOMMU used the same per-irq data structure as the Intel
implementation. Now that support for the AMD specific data
structure is upstream we can use this one instead.
Reviewed-by: NShuah Khan <shuahkhan@gmail.com>
Signed-off-by: NJoerg Roedel <joro@8bytes.org>

This function was intended as a fall-back if the map_sg
function is called for a device not mapped by the IOMMU.
Since the AMD IOMMU driver uses per-device dma_ops this can
never happen. So this function isn't needed anymore.
Reviewed-by: NShuah Khan <shuahkhan@gmail.com>
Signed-off-by: NJoerg Roedel <joro@8bytes.org>

This is required in case of PAMU, as it can support a window size of up
to 64G (even on 32bit).
Signed-off-by: NVarun Sethi <Varun.Sethi@freescale.com>
Signed-off-by: NJoerg Roedel <joro@8bytes.org>

Current driver does not clear the IOMMU event log interrupt bit
in the IOMMU status register after processing an interrupt.
This causes the IOMMU hardware to generate event log interrupt only once.
This has been observed in both IOMMU v1 and V2 hardware.
This patch clears the bit by writing 1 to bit 1 of the IOMMU
status register (MMIO Offset 2020h)
Signed-off-by: NSuravee Suthikulpanit <suravee.suthikulpanit@amd.com>
Signed-off-by: NJoerg Roedel <joro@8bytes.org>

There is a bug introduced with commit 27c2127a that causes
devices which are hot unplugged and then hot-replugged to
not have per-device dma_ops set. This causes these devices
to not function correctly. Fixed with this patch.

Cc: stable@vger.kernel.org
Reported-by: NAndreas Degert <andreas.degert@googlemail.com>
Signed-off-by: NJoerg Roedel <joro@8bytes.org>

Change to remove calc_devid() and use PCI_DEVID() from PCI instead.
Signed-off-by: NShuah Khan <shuah.khan@hp.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Acked-by: NJoerg Roedel <joro@8bytes.org>

Change to remove local PCI_BUS() define and use the new PCI_BUS_NUM()
interface from PCI.
Signed-off-by: NShuah Khan <shuah.khan@hp.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Acked-by: NJoerg Roedel <joro@8bytes.org>

dma_ops_domain_free on a NULL pointer is a no-op, so the NULL check in
amd_iommu_init_dma_ops() can be removed.
Signed-off-by: NCyril Roelandt <tipecaml@gmail.com>
Signed-off-by: NJoerg Roedel <joro@8bytes.org>

Move all the code to either to the header file
asm/irq_remapping.h or to drivers/iommu/.
Signed-off-by: NJoerg Roedel <joro@8bytes.org>
Acked-by: NSebastian Andrzej Siewior <sebastian@breakpoint.cc>
Reviewed-by: NKonrad Rzeszutek Wilk <konrad.wilk@oracle.com>

The AMD IOMMU driver only uses the page-sizes it gets from
IOMMU core and uses the appropriate page-size. So this
comment is not necessary.
Signed-off-by: NJoerg Roedel <joro@8bytes.org>

There is a bug in the hardware that will be triggered when
this page size is used. Make sure this does not happen.
Signed-off-by: NJoerg Roedel <joro@8bytes.org>

An alias doesn't always point to a physical device.  When this
happens we must first verify that the IOMMU group isn't rooted in
a device above the alias.  In this case the alias is effectively
just another quirk for the devices aliased to it.  Alternatively,
the virtual alias itself may be the root of the IOMMU group.  To
support this, allow a group to be hosted on the alias dev_data
for use by anything that might have the same alias.
Signed-off-by: NAlex williamson <alex.williamson@redhat.com>
Signed-off-by: NJoerg Roedel <joerg.roedel@amd.com>

Add a WARN_ON to make it clear why we don't add dma_pdev->dev to the
group we're allocating.
Signed-off-by: NAlex Williamson <alex.williamson@redhat.com>
Signed-off-by: NJoerg Roedel <joerg.roedel@amd.com>

Signed-off-by: NAlex Williamson <alex.williamson@redhat.com>
Signed-off-by: NJoerg Roedel <joerg.roedel@amd.com>

Signed-off-by: NAlex Williamson <alex.williamson@redhat.com>
Signed-off-by: NJoerg Roedel <joerg.roedel@amd.com>

This needs to be broken apart, start with pulling all the IOMMU
group init code into a new function.
Signed-off-by: NAlex Williamson <alex.williamson@redhat.com>
Signed-off-by: NJoerg Roedel <joerg.roedel@amd.com>

IRQ_DELAYED_DISABLE does not exist anymore. So this comment
is obsolete.
Signed-off-by: NJoerg Roedel <joerg.roedel@amd.com>

We should return NULL on error instead of the freed pointer.
Signed-off-by: NDan Carpenter <dan.carpenter@oracle.com>
Signed-off-by: NJoerg Roedel <joerg.roedel@amd.com>

Report the availability of irq remapping through the
IOMMU-API to allow KVM device passthrough again without
additional module parameter overrides.
Signed-off-by: NJoerg Roedel <joerg.roedel@amd.com>

Add the six routines required to setup interrupt remapping
with the AMD IOMMU. Also put it all together into the AMD
specific irq_remap_ops.
Signed-off-by: NJoerg Roedel <joerg.roedel@amd.com>

Add a routine to setup a HPET MSI interrupt for remapping.
Signed-off-by: NJoerg Roedel <joerg.roedel@amd.com>

Add routines to setup interrupt remapping for MSI
interrupts.
Signed-off-by: NJoerg Roedel <joerg.roedel@amd.com>

Add the routine to setup interrupt remapping for ioapic
interrupts. Also add a routine to change the affinity of an
irq and to free an irq allocation for interrupt remapping.
The last two functions will also be used for MSI interrupts.
Signed-off-by: NJoerg Roedel <joerg.roedel@amd.com>

Add routines to:

* Alloc remapping tables and single entries from these
  tables
* Change entries in the tables
* Free entries in the table
Signed-off-by: NJoerg Roedel <joerg.roedel@amd.com>

Add routine to invalidate the IOMMU cache for interupt
translations. Also include the IRTE caches when flushing all
IOMMU caches.
Signed-off-by: NJoerg Roedel <joerg.roedel@amd.com>

The irq remapping tables for the AMD IOMMU need to be
aligned on a 128 byte boundary. Create a seperate slab-cache
to guarantee this alignment.
Signed-off-by: NJoerg Roedel <joerg.roedel@amd.com>

The IVRS ACPI table provides information about the IOAPICs
and the HPETs available in the system and which PCI device
ID they use in transactions. Save that information for later
usage in interrupt remapping.
Signed-off-by: NJoerg Roedel <joerg.roedel@amd.com>
Reviewed-by: NKonrad Rzeszutek Wilk <konrad.wilk@oracle.com>

The new IOMMU groups code in the AMD IOMMU driver makes the
assumption that there is a pci_dev struct available for all
device-ids listed in the IVRS ACPI table. Unfortunatly this
assumption is not true and so this code causes a NULL
pointer dereference at boot on some systems.

Fix it by making sure the given pointer is never NULL when
passed to the group specific code. The real fix is larger
and will be queued for v3.7.
Reported-by: NFlorian Dazinger <florian@dazinger.net>
Signed-off-by: NJoerg Roedel <joerg.roedel@amd.com>

Fix some typos in comments and user-visible messages. No
functional changes.
Signed-off-by: NFrank Arnold <frank.arnold@amd.com>
Signed-off-by: NJoerg Roedel <joerg.roedel@amd.com>

SR-IOV can create buses without a bridge.  There may be other cases
where this happens as well.  In these cases skip to the parent bus
and continue testing devices there.
Signed-off-by: NAlex Williamson <alex.williamson@redhat.com>
Signed-off-by: NJoerg Roedel <joerg.roedel@amd.com>