Some AMD systems also have non-PCI devices which can do DMA.
Those can't be handled by the AMD IOMMU, as the hardware can
only handle PCI. These devices would end up with no dma_ops,
as neither the per-device nor the global dma_ops will get
set. SWIOTLB provides global dma_ops when it is active, so
make sure there are global dma_ops too when swiotlb is
disabled.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

In passthrough mode (iommu=pt) all devices are identity
mapped. If a device does not support 64bit DMA it might
still need remapping. Make sure swiotlb is initialized to
provide this remapping.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Since devices with IOMMUv2 functionality might be in the
same group as devices without it, allow those devices in
IOMMUv2 domains too.
Otherwise attaching the group with the IOMMUv2 device to the
domain will fail.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Remove the AMD IOMMU driver implementation for passthrough
mode and rely on the new iommu core features for that.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

This function contains the common parts between the
initialization of dma_ops_domains and usual protection
domains. This also fixes a long-standing bug which was
uncovered by recent changes, in which the api_lock was not
initialized for dma_ops_domains.
Reported-by: NGeorge Wang <xuw2015@gmail.com>
Tested-by: NGeorge Wang <xuw2015@gmail.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Make sure that we are skipping over large PTEs while walking
the page-table tree.

Cc: stable@kernel.org
Fixes: 5c34c403 ("iommu/amd: Fix memory leak in free_pagetable")
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Without this patch only -ENOTSUPP is handled, but there are
other possible errors. Handle them too.
Reported-by: NDan Carpenter <dan.carpenter@oracle.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

This function can fail. Propagate any errors back to the
initialization state machine.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

The list_head and target_dev members are not used anymore.
Remove them.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

With device intialization done in the add_device call-back
now there is no reason for this function anymore.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

A device that might be used for HSA needs to be in a direct
mapped domain so that all DMA-API mappings stay alive when
the IOMMUv2 stack is used.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Add support to allocate direct mapped domains through the
IOMMU-API.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

This enables allocation of DMA-API default domains from the
IOMMU core and switches allocation of domain dma-api domain
to the IOMMU core too.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Implement these two iommu-ops call-backs to make use of the
initialization and notifier features of the iommu core.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Add the get_dm_regions and put_dm_regions callbacks to the
iommu_ops of the AMD IOMMU driver.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

This reverts commit 5fc872c7.

The DMA-API does not strictly require that the memory
returned by dma_alloc_coherent is zeroed out. For that
another function (dma_zalloc_coherent) should be used. But
all other x86 DMA-API implementation I checked zero out the
memory, so that some drivers rely on it and break when it is
not.

It seems the (driver-)world is not yet ready for this
change, so revert it.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Handle this case to make sure boundary_size does not become
0 and trigger a BUG_ON later.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Move check of cfg->move_in_progress into send_cleanup_vector() to
prepare for simplifying struct irq_cfg.
Signed-off-by: NJiang Liu <jiang.liu@linux.intel.com>
Tested-by: NJoerg Roedel <jroedel@suse.de>
Cc: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Cc: David Cohen <david.a.cohen@linux.intel.com>
Cc: Sander Eikelenboom <linux@eikelenboom.it>
Cc: David Vrabel <david.vrabel@citrix.com>
Cc: Tony Luck <tony.luck@intel.com>
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Cc: iommu@lists.linux-foundation.org
Cc: Bjorn Helgaas <bhelgaas@google.com>
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
Cc: Randy Dunlap <rdunlap@infradead.org>
Cc: Yinghai Lu <yinghai@kernel.org>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Dimitri Sivanich <sivanich@sgi.com>
Cc: Joerg Roedel <joro@8bytes.org>
Link: http://lkml.kernel.org/r/1428978610-28986-26-git-send-email-jiang.liu@linux.intel.comSigned-off-by: NThomas Gleixner <tglx@linutronix.de>

Now only amd_iommu.c access irq_2_irte, so move it from hw_irq.h into
amd_iommu.c.
Signed-off-by: NJiang Liu <jiang.liu@linux.intel.com>
Tested-by: NJoerg Roedel <jroedel@suse.de>
Cc: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Cc: David Cohen <david.a.cohen@linux.intel.com>
Cc: Sander Eikelenboom <linux@eikelenboom.it>
Cc: David Vrabel <david.vrabel@citrix.com>
Cc: Tony Luck <tony.luck@intel.com>
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Cc: iommu@lists.linux-foundation.org
Cc: Bjorn Helgaas <bhelgaas@google.com>
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
Cc: Randy Dunlap <rdunlap@infradead.org>
Cc: Yinghai Lu <yinghai@kernel.org>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Dimitri Sivanich <sivanich@sgi.com>
Cc: Joerg Roedel <joro@8bytes.org>
Link: http://lkml.kernel.org/r/1428978610-28986-16-git-send-email-jiang.liu@linux.intel.comSigned-off-by: NThomas Gleixner <tglx@linutronix.de>

Now there is no user of irq_cfg.irq_remapped, so remove it.
Signed-off-by: NJiang Liu <jiang.liu@linux.intel.com>
Tested-by: NJoerg Roedel <jroedel@suse.de>
Cc: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Cc: David Cohen <david.a.cohen@linux.intel.com>
Cc: Sander Eikelenboom <linux@eikelenboom.it>
Cc: David Vrabel <david.vrabel@citrix.com>
Cc: Tony Luck <tony.luck@intel.com>
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Cc: iommu@lists.linux-foundation.org
Cc: Bjorn Helgaas <bhelgaas@google.com>
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
Cc: Randy Dunlap <rdunlap@infradead.org>
Cc: Yinghai Lu <yinghai@kernel.org>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Dimitri Sivanich <sivanich@sgi.com>
Cc: Joerg Roedel <joro@8bytes.org>
Link: http://lkml.kernel.org/r/1428978610-28986-14-git-send-email-jiang.liu@linux.intel.comSigned-off-by: NThomas Gleixner <tglx@linutronix.de>

Now we have converted to hierarchical irqdomains, so clean up unused
code.
Signed-off-by: NJiang Liu <jiang.liu@linux.intel.com>
Tested-by: NJoerg Roedel <jroedel@suse.de>
Cc: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Cc: David Cohen <david.a.cohen@linux.intel.com>
Cc: Sander Eikelenboom <linux@eikelenboom.it>
Cc: David Vrabel <david.vrabel@citrix.com>
Cc: Tony Luck <tony.luck@intel.com>
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Cc: iommu@lists.linux-foundation.org
Cc: Bjorn Helgaas <bhelgaas@google.com>
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
Cc: Randy Dunlap <rdunlap@infradead.org>
Cc: Yinghai Lu <yinghai@kernel.org>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Dimitri Sivanich <sivanich@sgi.com>
Cc: Joerg Roedel <joro@8bytes.org>
Link: http://lkml.kernel.org/r/1428978610-28986-12-git-send-email-jiang.liu@linux.intel.comSigned-off-by: NThomas Gleixner <tglx@linutronix.de>

Now MSI interrupt has been converted to new hierarchical irqdomain
interfaces, so remove legacy MSI related code.
Signed-off-by: NJiang Liu <jiang.liu@linux.intel.com>
Acked-by: NJoerg Roedel <jroedel@suse.de>
Cc: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Cc: David Cohen <david.a.cohen@linux.intel.com>
Cc: Sander Eikelenboom <linux@eikelenboom.it>
Cc: David Vrabel <david.vrabel@citrix.com>
Cc: Tony Luck <tony.luck@intel.com>
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Cc: iommu@lists.linux-foundation.org
Cc: Bjorn Helgaas <bhelgaas@google.com>
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
Cc: Randy Dunlap <rdunlap@infradead.org>
Cc: Yinghai Lu <yinghai@kernel.org>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Dimitri Sivanich <sivanich@sgi.com>
Cc: Joerg Roedel <joro@8bytes.org>
Link: http://lkml.kernel.org/r/1428905519-23704-17-git-send-email-jiang.liu@linux.intel.comSigned-off-by: NThomas Gleixner <tglx@linutronix.de>

Signed-off-by: NJiang Liu <jiang.liu@linux.intel.com>
Acked-by: NJoerg Roedel <jroedel@suse.de>
Cc: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Cc: David Cohen <david.a.cohen@linux.intel.com>
Cc: Sander Eikelenboom <linux@eikelenboom.it>
Cc: David Vrabel <david.vrabel@citrix.com>
Cc: Tony Luck <tony.luck@intel.com>
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Cc: iommu@lists.linux-foundation.org
Cc: Bjorn Helgaas <bhelgaas@google.com>
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
Cc: Randy Dunlap <rdunlap@infradead.org>
Cc: Yinghai Lu <yinghai@kernel.org>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Dimitri Sivanich <sivanich@sgi.com>
Cc: Joerg Roedel <joro@8bytes.org>
Link: http://lkml.kernel.org/r/1428905519-23704-12-git-send-email-jiang.liu@linux.intel.comSigned-off-by: NThomas Gleixner <tglx@linutronix.de>

When a default page-size for given level should be mapped,
the level encoding must be 0 rather than 7. This fixes an
issue seen on IOMMUv2 hardware, where this encoding is
enforced.
Tested-by: NSuravee Suthikulpanit <Suravee.Suthikulpanit@amd.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Now that fetch_pte returns the page-size of the pte, this
function can be optimized too.
Tested-by: NSuravee Suthikulpanit <Suravee.Suthikulpanit@amd.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Now that fetch_pte returns the page-size of the pte, the
call in this function can also be optimized a little bit.
Tested-by: NSuravee Suthikulpanit <Suravee.Suthikulpanit@amd.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Now that fetch_pte returns the page-size of the pte, this
function can be optimized a lot.
Tested-by: NSuravee Suthikulpanit <Suravee.Suthikulpanit@amd.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Extend the fetch_pte function to also return the page-size
that is mapped by the returned pte.
Tested-by: NSuravee Suthikulpanit <Suravee.Suthikulpanit@amd.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Add code to allocate memory from the contiguous memory
allocator to support coherent allocations larger than 8MB.
Tested-by: NSuravee Suthikulpanit <Suravee.Suthikulpanit@amd.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Don't explicitly add __GFP_ZERO to the allocator flags.
Leave this up to the caller.
Tested-by: NSuravee Suthikulpanit <Suravee.Suthikulpanit@amd.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Detaching a device from its domain at this event is
problematic for several reasons:

	* The device might me in an alias group and
	  detaching it will also detach all other devices in
	  the group. This removes valid DMA mappings from
	  the other devices causing io-page-faults and lets
	  these devices fail.

	* Devices might have unity mappings specified by the
	  IVRS table. These mappings are required for the
	  device even when no device driver is attached.
	  Detaching the device from its domain in driver
	  unbind will also remove these unity mappings.

This patch removes the handling of the BUS_NOTIFY_UNBOUND_DRIVER
event to prevent these issues and align it better with the
behavior of the VT-d driver.
Tested-by: NSuravee Suthikulpanit <Suravee.Suthikulpanit@amd.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Use the new device-notifier event instead of the old
BUS_NOTIFY_DEL_DEVICE to make sure the device driver had a
chance to uninit the device before all its mappings are
teared down.
Tested-by: NSuravee Suthikulpanit <Suravee.Suthikulpanit@amd.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Implement the new iommu-ops function pointers and remove the
obsolete domain_init and domain_destroy functions.
Reviewed-by: NAlex Williamson <alex.williamson@redhat.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

This patch was produced using Coccinelle. A simplified version of the
semantic patch is:

@r exists@
identifier f;
local idexpression u8 x;
identifier xname;
@@

f(...) {
...when any
(
  x@xname = 1;
|
  x@xname = 0;
)
...when any
}

@bad exists@
identifier r.f;
local idexpression u8 r.x
expression e1 != {0, 1}, e2;
@@

f(...) {
...when any
(
  x = e1;
|
  x + e2
)
...when any
}

@depends on !bad@
identifier r.f;
local idexpression u8 r.x;
identifier r.xname;
@@

f(...) {
...
++ bool xname;
- int xname;
<...
(
  x =
- 1
+ true
|
  x =
- -1
+ false
)
...>

}
Signed-off-by: NQuentin Lambert <lambert.quentin@gmail.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

The AMD address is dead for a long time already, replace it
with a working one.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Simplify irq_remapping code by killing irq_remapping_supported() and
related interfaces.

Joerg posted a similar patch at https://lkml.org/lkml/2014/12/15/490,
so assume an signed-off from Joerg.
Signed-off-by: NJiang Liu <jiang.liu@linux.intel.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>
Tested-by: NJoerg Roedel <joro@8bytes.org>
Cc: Tony Luck <tony.luck@intel.com>
Cc: iommu@lists.linux-foundation.org
Cc: H. Peter Anvin <hpa@linux.intel.com>
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: Yinghai Lu <yinghai@kernel.org>
Cc: Borislav Petkov <bp@alien8.de>
Cc: David Rientjes <rientjes@google.com>
Cc: HATAYAMA Daisuke <d.hatayama@jp.fujitsu.com>
Cc: Jan Beulich <JBeulich@suse.com>
Cc: Richard Weinberger <richard@nod.at>
Cc: Oren Twaig <oren@scalemp.com>
Link: http://lkml.kernel.org/r/1420615903-28253-14-git-send-email-jiang.liu@linux.intel.comSigned-off-by: NThomas Gleixner <tglx@linutronix.de>

Use helpers to access irq_cfg data structure associated with IRQ,
instead of accessing irq_data->chip_data directly. Later we can
rewrite those helpers to support hierarchy irqdomain.
Signed-off-by: NJiang Liu <jiang.liu@linux.intel.com>
Cc: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Cc: Tony Luck <tony.luck@intel.com>
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Cc: iommu@lists.linux-foundation.org
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
Cc: Bjorn Helgaas <bhelgaas@google.com>
Cc: Randy Dunlap <rdunlap@infradead.org>
Cc: Yinghai Lu <yinghai@kernel.org>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Joerg Roedel <joro@8bytes.org>
Link: http://lkml.kernel.org/r/1414397531-28254-20-git-send-email-jiang.liu@linux.intel.comSigned-off-by: NThomas Gleixner <tglx@linutronix.de>

Some versions of GCC get unduly upset when confronted with a switch
that doesn't explicitly handle all cases of an enum, despite having an
implicit default case following the actualy switch statement:

   drivers/iommu/amd_iommu.c: In function 'amd_iommu_capable':
>> drivers/iommu/amd_iommu.c:3409:2: warning: enumeration value 'IOMMU_CAP_NOEXEC' not handled in switch [-Wswitch]
     switch (cap) {

This patch adds a case for IOMMU_CAP_NOEXEC to the amd IOMMU driver to
remove this warning.

Cc: Joerg Roedel <jroedel@suse.de>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

Mapping and unmapping are more often than not in the critical path.
map_sg allows IOMMU driver implementations to optimize the process
of mapping buffers into the IOMMU page tables.

Instead of mapping a buffer one page at a time and requiring potentially
expensive TLB operations for each page, this function allows the driver
to map all pages in one go and defer TLB maintenance until after all
pages have been mapped.

Additionally, the mapping operation would be faster in general since
clients does not have to keep calling map API over and over again for
each physically contiguous chunk of memory that needs to be mapped to a
virtually contiguous region.
Signed-off-by: NOlav Haugan <ohaugan@codeaurora.org>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>