The struct is only used there, so it doesn't need to be in
the header file.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

When multiple devices are detached in __detach_device, they
are also removed from the domains dev_list. This makes it
unsafe to use list_for_each_entry_safe, as the next pointer
might also not be in the list anymore after __detach_device
returns. So just repeatedly remove the first element of the
list until it is empty.

Cc: stable@vger.kernel.org
Tested-by: NMarti Raudsepp <marti@juffo.org>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

This structure is read-only data and should never be modified.
Signed-off-by: NThierry Reding <treding@nvidia.com>
Signed-off-by: NGreg Kroah-Hartman <gregkh@linuxfoundation.org>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

AMD-Vi support for IOMMU sysfs.  This allows us to associate devices
with a specific IOMMU device and examine the capabilities and features
of that IOMMU.  The AMD IOMMU is hosted on and actual PCI device, so
we make that device the parent for the IOMMU class device.  This
initial implementaiton exposes only the capability header and extended
features register for the IOMMU.

# find /sys | grep ivhd
/sys/devices/pci0000:00/0000:00:00.2/iommu/ivhd0
/sys/devices/pci0000:00/0000:00:00.2/iommu/ivhd0/devices
/sys/devices/pci0000:00/0000:00:00.2/iommu/ivhd0/devices/0000:00:00.0
/sys/devices/pci0000:00/0000:00:00.2/iommu/ivhd0/devices/0000:00:02.0
/sys/devices/pci0000:00/0000:00:00.2/iommu/ivhd0/devices/0000:00:04.0
/sys/devices/pci0000:00/0000:00:00.2/iommu/ivhd0/devices/0000:00:09.0
/sys/devices/pci0000:00/0000:00:00.2/iommu/ivhd0/devices/0000:00:11.0
/sys/devices/pci0000:00/0000:00:00.2/iommu/ivhd0/devices/0000:00:12.0
/sys/devices/pci0000:00/0000:00:00.2/iommu/ivhd0/devices/0000:00:12.2
/sys/devices/pci0000:00/0000:00:00.2/iommu/ivhd0/devices/0000:00:13.0
...
/sys/devices/pci0000:00/0000:00:00.2/iommu/ivhd0/power
/sys/devices/pci0000:00/0000:00:00.2/iommu/ivhd0/power/control
...
/sys/devices/pci0000:00/0000:00:00.2/iommu/ivhd0/device
/sys/devices/pci0000:00/0000:00:00.2/iommu/ivhd0/subsystem
/sys/devices/pci0000:00/0000:00:00.2/iommu/ivhd0/amd-iommu
/sys/devices/pci0000:00/0000:00:00.2/iommu/ivhd0/amd-iommu/cap
/sys/devices/pci0000:00/0000:00:00.2/iommu/ivhd0/amd-iommu/features
/sys/devices/pci0000:00/0000:00:00.2/iommu/ivhd0/uevent
/sys/class/iommu/ivhd0
Signed-off-by: NAlex Williamson <alex.williamson@redhat.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

The common iommu_group_get_for_dev() allows us to greatly simplify
our group lookup for a new device.  Also, since we insert IVRS
aliases into the PCI DMA alias quirks, we should alway come up with
the same results as the existing code.
Signed-off-by: NAlex Williamson <alex.williamson@redhat.com>
Cc: Joerg Roedel <joro@8bytes.org>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

AMD-Vi already has a concept of an alias provided via the IVRS table.
Now that PCI-core also understands aliases, we need to incorporate
both aspects when programming the IOMMU.  IVRS is generally quite
reliable, so we continue to prefer it when an alias is present.  For
cases where we have an IVRS alias that does not match the PCI alias
or where PCI does not report an alias, report the mismatch to allow
us to collect more quirks and dynamically incorporate the alias into
the device alias quirks where possible.

This should allow AMD-Vi to work with devices like Marvell and Ricoh
with DMA function alias quirks unknown to the BIOS.
Signed-off-by: NAlex Williamson <alex.williamson@redhat.com>
Cc: Joerg Roedel <joro@8bytes.org>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Fix two compile warnings about unused variables introduced
by commit ecef115d.
Reported-by: NPaul Bolle <pebolle@tiscali.nl>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

amd_iommu_rlookup_table[devid] != NULL is already guaranteed
by check_device called before, it's fine to attach device at
this point.
Signed-off-by: NVaughan Cao <vaughan.cao@oracle.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

An apparent cut and paste error prevents the correct flags from being
set on the alias device resulting in MSI on conventional PCI devices
failing to work.  This also produces error events from the IOMMU like:

AMD-Vi: Event logged [INVALID_DEVICE_REQUEST device=00:14.4 address=0x000000fdf8000000 flags=0x0a00]

Where 14.4 is a PCIe-to-PCI bridge with a device behind it trying to
use MSI interrupts.
Signed-off-by: NAlex Williamson <alex.williamson@redhat.com>
Cc: stable@vger.kernel.org
Signed-off-by: NJoerg Roedel <joro@8bytes.org>

In reality, the spec can only support 16-bit PASID since
INVALIDATE_IOTLB_PAGES and COMPLETE_PPR_REQUEST commands only allow 16-bit
PASID. So, we updated the PASID_MASK accordingly and invoke BUG_ON
if the hardware is reporting PASmax more than 16-bit.

Besides, max PASID is defined as ((2^(PASmax+1)) - 1). The current does not
determine this correctly.
Signed-off-by: NSuravee Suthikulpanit <suravee.suthikulpanit@amd.com>
Tested-by: NJay Cornwall <Jay.Cornwall@amd.com>
Signed-off-by: NJoerg Roedel <joro@8bytes.org>

This patch corrects the PASID format in the INVALIDATE_IOTLB_PAGES
command, which was caused by incorrect information in
the AMD IOMMU Architectural Specification v2.01 document.

    Incorrect format:
         cmd->data[0][16:23] = PASID[7:0]
         cmd->data[1][16:27] = PASID[19:8]

     Correct format:
         cmd->data[0][16:23] = PASID[15:8]
         cmd->data[1][16:23] = PASID[7:0]

However, this does not affect the IOMMUv2 hardware implementation,
and has been corrected since version 2.02 of the specification
(available through AMD NDA).
Signed-off-by: NJay Cornwall <jay.cornwall@amd.com>
Reviewed-by: NSuravee Suthikulpanit <suravee.suthikulpanit@amd.com>
Signed-off-by: NJoerg Roedel <joro@8bytes.org>

Use PCI standard marco dev_is_pci() instead of directly compare
pci_bus_type to check whether it is pci device.
Signed-off-by: NYijing Wang <wangyijing@huawei.com>
Signed-off-by: NJoerg Roedel <joro@8bytes.org>

Detected by cppcheck.
Signed-off-by: NKamil Dudka <kdudka@redhat.com>
Signed-off-by: NJoerg Roedel <joro@8bytes.org>

If we use a large mapping, the expectation is that only unmaps from
the first pte in the superpage are supported.  Unmaps from offsets
into the superpage should fail (ie. return zero sized unmap).  In the
current code, unmapping from an offset clears the size of the full
mapping starting from an offset.  For instance, if we map a 16k
physically contiguous range at IOVA 0x0 with a large page, then
attempt to unmap 4k at offset 12k, 4 ptes are cleared (12k - 28k) and
the unmap returns 16k unmapped.  This potentially incorrectly clears
valid mappings and confuses drivers like VFIO that use the unmap size
to release pinned pages.

Fix by refusing to unmap from offsets into the page.
Signed-off-by: NAlex Williamson <alex.williamson@redhat.com>
Cc: stable@vger.kernel.org
Signed-off-by: NJoerg Roedel <joro@8bytes.org>

The IOMMU pagetables can have up to 6 levels, but the code
in free_pagetable() only releases the first 3 levels. Fix
this leak by releasing all levels.
Reported-by: NAlex Williamson <alex.williamson@redhat.com>
Signed-off-by: NJoerg Roedel <joro@8bytes.org>
Reviewed-by: NAlex Williamson <alex.williamson@redhat.com>

If a device is multifunction and does not have ACS enabled then we
assume that the entire package lacks ACS and use function 0 as the
base of the group.  The PCIe spec however states that components are
permitted to implement ACS on some, none, or all of their applicable
functions.  It's therefore conceivable that function 0 may be fully
independent and support ACS while other functions do not.  Instead
use the lowest function of the slot that does not have ACS enabled
as the base of the group.  This may be the current device, which is
intentional.  So long as we use a consistent algorithm, all the
non-ACS functions will be grouped together and ACS functions will
get separate groups.
Signed-off-by: NAlex Williamson <alex.williamson@redhat.com>
Signed-off-by: NJoerg Roedel <joro@8bytes.org>

The swap_pci_ref function is used by the IOMMU API code for
swapping pci device pointers, while determining the iommu
group for the device.
Currently this function was being implemented for different
IOMMU drivers.  This patch moves the function to a new file,
drivers/iommu/pci.h so that the implementation can be
shared across various IOMMU drivers.
Signed-off-by: NVarun Sethi <Varun.Sethi@freescale.com>
Signed-off-by: NJoerg Roedel <joro@8bytes.org>

In the current interrupt handling scheme, there are as many threads as
the number of IOMMUs. Each thread is created and assigned to an IOMMU at
the time of registering interrupt handlers (request_threaded_irq).
When an IOMMU HW generates an interrupt, the irq handler (top half) wakes up
the corresponding thread to process event and PPR logs of all IOMMUs
starting from the 1st IOMMU.

In the system with multiple IOMMU,this handling scheme complicates the
synchronization of the IOMMU data structures and status registers as
there could be multiple threads competing for the same IOMMU while
the other IOMMU could be left unhandled.

To simplify, this patch is proposing a different interrupt handling scheme
by having each thread only managing interrupts of the corresponding IOMMU.
This can be achieved by passing the struct amd_iommu when registering the
interrupt handlers. This structure is unique for each IOMMU and can be used
by the bottom half thread to identify the IOMMU to be handled instead
of calling for_each_iommu.  Besides this also eliminate the needs to lock
the IOMMU for processing event and PPR logs.
Signed-off-by: NSuravee Suthikulpanit <suravee.suthikulpanit@amd.com>
Signed-off-by: NJoerg Roedel <joro@8bytes.org>

Work around an IOMMU  hardware bug where clearing the
EVT_INT or PPR_INT bit in the status register may race with
the hardware trying to set it again. When not handled the
bit might not be cleared and we lose all future event or ppr
interrupts.
Reported-by: NSuravee Suthikulpanit <suravee.suthikulpanit@amd.com>
Cc: stable@vger.kernel.org
Signed-off-by: NJoerg Roedel <joro@8bytes.org>

Fixes a lockdep warning.

Cc: stable@vger.kernel.org # >= v3.7
Reviewed-by: NShuah Khan <shuahkhan@gmail.com>
Signed-off-by: NJoerg Roedel <joro@8bytes.org>

For compatibility reasons the irq remapping code for the AMD
IOMMU used the same per-irq data structure as the Intel
implementation. Now that support for the AMD specific data
structure is upstream we can use this one instead.
Reviewed-by: NShuah Khan <shuahkhan@gmail.com>
Signed-off-by: NJoerg Roedel <joro@8bytes.org>

This function was intended as a fall-back if the map_sg
function is called for a device not mapped by the IOMMU.
Since the AMD IOMMU driver uses per-device dma_ops this can
never happen. So this function isn't needed anymore.
Reviewed-by: NShuah Khan <shuahkhan@gmail.com>
Signed-off-by: NJoerg Roedel <joro@8bytes.org>

This is required in case of PAMU, as it can support a window size of up
to 64G (even on 32bit).
Signed-off-by: NVarun Sethi <Varun.Sethi@freescale.com>
Signed-off-by: NJoerg Roedel <joro@8bytes.org>

Current driver does not clear the IOMMU event log interrupt bit
in the IOMMU status register after processing an interrupt.
This causes the IOMMU hardware to generate event log interrupt only once.
This has been observed in both IOMMU v1 and V2 hardware.
This patch clears the bit by writing 1 to bit 1 of the IOMMU
status register (MMIO Offset 2020h)
Signed-off-by: NSuravee Suthikulpanit <suravee.suthikulpanit@amd.com>
Signed-off-by: NJoerg Roedel <joro@8bytes.org>

There is a bug introduced with commit 27c2127a that causes
devices which are hot unplugged and then hot-replugged to
not have per-device dma_ops set. This causes these devices
to not function correctly. Fixed with this patch.

Cc: stable@vger.kernel.org
Reported-by: NAndreas Degert <andreas.degert@googlemail.com>
Signed-off-by: NJoerg Roedel <joro@8bytes.org>

Change to remove calc_devid() and use PCI_DEVID() from PCI instead.
Signed-off-by: NShuah Khan <shuah.khan@hp.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Acked-by: NJoerg Roedel <joro@8bytes.org>

Change to remove local PCI_BUS() define and use the new PCI_BUS_NUM()
interface from PCI.
Signed-off-by: NShuah Khan <shuah.khan@hp.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Acked-by: NJoerg Roedel <joro@8bytes.org>

dma_ops_domain_free on a NULL pointer is a no-op, so the NULL check in
amd_iommu_init_dma_ops() can be removed.
Signed-off-by: NCyril Roelandt <tipecaml@gmail.com>
Signed-off-by: NJoerg Roedel <joro@8bytes.org>

Move all the code to either to the header file
asm/irq_remapping.h or to drivers/iommu/.
Signed-off-by: NJoerg Roedel <joro@8bytes.org>
Acked-by: NSebastian Andrzej Siewior <sebastian@breakpoint.cc>
Reviewed-by: NKonrad Rzeszutek Wilk <konrad.wilk@oracle.com>

The AMD IOMMU driver only uses the page-sizes it gets from
IOMMU core and uses the appropriate page-size. So this
comment is not necessary.
Signed-off-by: NJoerg Roedel <joro@8bytes.org>

There is a bug in the hardware that will be triggered when
this page size is used. Make sure this does not happen.
Signed-off-by: NJoerg Roedel <joro@8bytes.org>

An alias doesn't always point to a physical device.  When this
happens we must first verify that the IOMMU group isn't rooted in
a device above the alias.  In this case the alias is effectively
just another quirk for the devices aliased to it.  Alternatively,
the virtual alias itself may be the root of the IOMMU group.  To
support this, allow a group to be hosted on the alias dev_data
for use by anything that might have the same alias.
Signed-off-by: NAlex williamson <alex.williamson@redhat.com>
Signed-off-by: NJoerg Roedel <joerg.roedel@amd.com>

Add a WARN_ON to make it clear why we don't add dma_pdev->dev to the
group we're allocating.
Signed-off-by: NAlex Williamson <alex.williamson@redhat.com>
Signed-off-by: NJoerg Roedel <joerg.roedel@amd.com>

Signed-off-by: NAlex Williamson <alex.williamson@redhat.com>
Signed-off-by: NJoerg Roedel <joerg.roedel@amd.com>

Signed-off-by: NAlex Williamson <alex.williamson@redhat.com>
Signed-off-by: NJoerg Roedel <joerg.roedel@amd.com>

This needs to be broken apart, start with pulling all the IOMMU
group init code into a new function.
Signed-off-by: NAlex Williamson <alex.williamson@redhat.com>
Signed-off-by: NJoerg Roedel <joerg.roedel@amd.com>

IRQ_DELAYED_DISABLE does not exist anymore. So this comment
is obsolete.
Signed-off-by: NJoerg Roedel <joerg.roedel@amd.com>

We should return NULL on error instead of the freed pointer.
Signed-off-by: NDan Carpenter <dan.carpenter@oracle.com>
Signed-off-by: NJoerg Roedel <joerg.roedel@amd.com>

Report the availability of irq remapping through the
IOMMU-API to allow KVM device passthrough again without
additional module parameter overrides.
Signed-off-by: NJoerg Roedel <joerg.roedel@amd.com>

Add the six routines required to setup interrupt remapping
with the AMD IOMMU. Also put it all together into the AMD
specific irq_remap_ops.
Signed-off-by: NJoerg Roedel <joerg.roedel@amd.com>