This function contains the common parts between the
initialization of dma_ops_domains and usual protection
domains. This also fixes a long-standing bug which was
uncovered by recent changes, in which the api_lock was not
initialized for dma_ops_domains.
Reported-by: NGeorge Wang <xuw2015@gmail.com>
Tested-by: NGeorge Wang <xuw2015@gmail.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Make sure that we are skipping over large PTEs while walking
the page-table tree.

Cc: stable@kernel.org
Fixes: 5c34c403 ("iommu/amd: Fix memory leak in free_pagetable")
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Without this patch only -ENOTSUPP is handled, but there are
other possible errors. Handle them too.
Reported-by: NDan Carpenter <dan.carpenter@oracle.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

This function can fail. Propagate any errors back to the
initialization state machine.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

The list_head and target_dev members are not used anymore.
Remove them.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

With device intialization done in the add_device call-back
now there is no reason for this function anymore.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

A device that might be used for HSA needs to be in a direct
mapped domain so that all DMA-API mappings stay alive when
the IOMMUv2 stack is used.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Add support to allocate direct mapped domains through the
IOMMU-API.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

This enables allocation of DMA-API default domains from the
IOMMU core and switches allocation of domain dma-api domain
to the IOMMU core too.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Implement these two iommu-ops call-backs to make use of the
initialization and notifier features of the iommu core.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Add the get_dm_regions and put_dm_regions callbacks to the
iommu_ops of the AMD IOMMU driver.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

This reverts commit 5fc872c7.

The DMA-API does not strictly require that the memory
returned by dma_alloc_coherent is zeroed out. For that
another function (dma_zalloc_coherent) should be used. But
all other x86 DMA-API implementation I checked zero out the
memory, so that some drivers rely on it and break when it is
not.

It seems the (driver-)world is not yet ready for this
change, so revert it.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Handle this case to make sure boundary_size does not become
0 and trigger a BUG_ON later.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

When a default page-size for given level should be mapped,
the level encoding must be 0 rather than 7. This fixes an
issue seen on IOMMUv2 hardware, where this encoding is
enforced.
Tested-by: NSuravee Suthikulpanit <Suravee.Suthikulpanit@amd.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Now that fetch_pte returns the page-size of the pte, this
function can be optimized too.
Tested-by: NSuravee Suthikulpanit <Suravee.Suthikulpanit@amd.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Now that fetch_pte returns the page-size of the pte, the
call in this function can also be optimized a little bit.
Tested-by: NSuravee Suthikulpanit <Suravee.Suthikulpanit@amd.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Now that fetch_pte returns the page-size of the pte, this
function can be optimized a lot.
Tested-by: NSuravee Suthikulpanit <Suravee.Suthikulpanit@amd.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Extend the fetch_pte function to also return the page-size
that is mapped by the returned pte.
Tested-by: NSuravee Suthikulpanit <Suravee.Suthikulpanit@amd.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Add code to allocate memory from the contiguous memory
allocator to support coherent allocations larger than 8MB.
Tested-by: NSuravee Suthikulpanit <Suravee.Suthikulpanit@amd.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Don't explicitly add __GFP_ZERO to the allocator flags.
Leave this up to the caller.
Tested-by: NSuravee Suthikulpanit <Suravee.Suthikulpanit@amd.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Detaching a device from its domain at this event is
problematic for several reasons:

	* The device might me in an alias group and
	  detaching it will also detach all other devices in
	  the group. This removes valid DMA mappings from
	  the other devices causing io-page-faults and lets
	  these devices fail.

	* Devices might have unity mappings specified by the
	  IVRS table. These mappings are required for the
	  device even when no device driver is attached.
	  Detaching the device from its domain in driver
	  unbind will also remove these unity mappings.

This patch removes the handling of the BUS_NOTIFY_UNBOUND_DRIVER
event to prevent these issues and align it better with the
behavior of the VT-d driver.
Tested-by: NSuravee Suthikulpanit <Suravee.Suthikulpanit@amd.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Use the new device-notifier event instead of the old
BUS_NOTIFY_DEL_DEVICE to make sure the device driver had a
chance to uninit the device before all its mappings are
teared down.
Tested-by: NSuravee Suthikulpanit <Suravee.Suthikulpanit@amd.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Implement the new iommu-ops function pointers and remove the
obsolete domain_init and domain_destroy functions.
Reviewed-by: NAlex Williamson <alex.williamson@redhat.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

This patch was produced using Coccinelle. A simplified version of the
semantic patch is:

@r exists@
identifier f;
local idexpression u8 x;
identifier xname;
@@

f(...) {
...when any
(
  x@xname = 1;
|
  x@xname = 0;
)
...when any
}

@bad exists@
identifier r.f;
local idexpression u8 r.x
expression e1 != {0, 1}, e2;
@@

f(...) {
...when any
(
  x = e1;
|
  x + e2
)
...when any
}

@depends on !bad@
identifier r.f;
local idexpression u8 r.x;
identifier r.xname;
@@

f(...) {
...
++ bool xname;
- int xname;
<...
(
  x =
- 1
+ true
|
  x =
- -1
+ false
)
...>

}
Signed-off-by: NQuentin Lambert <lambert.quentin@gmail.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

The AMD address is dead for a long time already, replace it
with a working one.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Simplify irq_remapping code by killing irq_remapping_supported() and
related interfaces.

Joerg posted a similar patch at https://lkml.org/lkml/2014/12/15/490,
so assume an signed-off from Joerg.
Signed-off-by: NJiang Liu <jiang.liu@linux.intel.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>
Tested-by: NJoerg Roedel <joro@8bytes.org>
Cc: Tony Luck <tony.luck@intel.com>
Cc: iommu@lists.linux-foundation.org
Cc: H. Peter Anvin <hpa@linux.intel.com>
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: Yinghai Lu <yinghai@kernel.org>
Cc: Borislav Petkov <bp@alien8.de>
Cc: David Rientjes <rientjes@google.com>
Cc: HATAYAMA Daisuke <d.hatayama@jp.fujitsu.com>
Cc: Jan Beulich <JBeulich@suse.com>
Cc: Richard Weinberger <richard@nod.at>
Cc: Oren Twaig <oren@scalemp.com>
Link: http://lkml.kernel.org/r/1420615903-28253-14-git-send-email-jiang.liu@linux.intel.comSigned-off-by: NThomas Gleixner <tglx@linutronix.de>

Use helpers to access irq_cfg data structure associated with IRQ,
instead of accessing irq_data->chip_data directly. Later we can
rewrite those helpers to support hierarchy irqdomain.
Signed-off-by: NJiang Liu <jiang.liu@linux.intel.com>
Cc: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Cc: Tony Luck <tony.luck@intel.com>
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Cc: iommu@lists.linux-foundation.org
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
Cc: Bjorn Helgaas <bhelgaas@google.com>
Cc: Randy Dunlap <rdunlap@infradead.org>
Cc: Yinghai Lu <yinghai@kernel.org>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Joerg Roedel <joro@8bytes.org>
Link: http://lkml.kernel.org/r/1414397531-28254-20-git-send-email-jiang.liu@linux.intel.comSigned-off-by: NThomas Gleixner <tglx@linutronix.de>

Some versions of GCC get unduly upset when confronted with a switch
that doesn't explicitly handle all cases of an enum, despite having an
implicit default case following the actualy switch statement:

   drivers/iommu/amd_iommu.c: In function 'amd_iommu_capable':
>> drivers/iommu/amd_iommu.c:3409:2: warning: enumeration value 'IOMMU_CAP_NOEXEC' not handled in switch [-Wswitch]
     switch (cap) {

This patch adds a case for IOMMU_CAP_NOEXEC to the amd IOMMU driver to
remove this warning.

Cc: Joerg Roedel <jroedel@suse.de>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

Mapping and unmapping are more often than not in the critical path.
map_sg allows IOMMU driver implementations to optimize the process
of mapping buffers into the IOMMU page tables.

Instead of mapping a buffer one page at a time and requiring potentially
expensive TLB operations for each page, this function allows the driver
to map all pages in one go and defer TLB maintenance until after all
pages have been mapped.

Additionally, the mapping operation would be faster in general since
clients does not have to keep calling map API over and over again for
each physically contiguous chunk of memory that needs to be mapped to a
virtually contiguous region.
Signed-off-by: NOlav Haugan <ohaugan@codeaurora.org>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Commit 71054d88 ("x86, hpet: Introduce x86_msi_ops.setup_hpet_msi")
introduced x86_msi_ops.setup_hpet_msi to setup hpet MSI irq
when irq remapping enabled. This caused a regression of
hpet MSI irq remapping.

Original code flow before commit 71054d88:
hpet_setup_msi_irq()
	arch_setup_hpet_msi()
		setup_hpet_msi_remapped()
			remap_ops->setup_hpet_msi()
				alloc_irte()
		msi_compose_msg()
		hpet_msi_write()
		...

Current code flow after commit 71054d88:
hpet_setup_msi_irq()
	x86_msi.setup_hpet_msi()
		setup_hpet_msi_remapped()
			intel_setup_hpet_msi()
				alloc_irte()

Currently, we only call alloc_irte() for hpet MSI, but
do not composed and wrote its msg...
Signed-off-by: NYijing Wang <wangyijing@huawei.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

For a PCI device, aliases from the IVRS table won't be populated
into dma_alias_devfn until after iommu_init_device() is called on
each device.  We therefore want to split init_iommu_group() to
be called from a separate loop immediately following.
Signed-off-by: NAlex Williamson <alex.williamson@redhat.com>
Cc: stable@vger.kernel.org # 3.17
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

under low memory conditions, alloc_pte() may return a NULL pointer.
iommu_map_page() does not check it and will panic the system.
Signed-off-by: NMaurizio Lombardi <mlombard@redhat.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Signed-off-by: NJoerg Roedel <jroedel@suse.de>

This reference count is not used anymore, as all devices in
an alias group are now attached and detached together.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Change tha device attach and detach semantic to apply to all
devices in an alias group. This means all devices in an
alias group are now attached and detached at the same time.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Some broken devices might use any request-id from the alias
group, so we need to set a DTE entry for every device in
there. This patch adds creation of those lists.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

The struct is only used there, so it doesn't need to be in
the header file.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

When multiple devices are detached in __detach_device, they
are also removed from the domains dev_list. This makes it
unsafe to use list_for_each_entry_safe, as the next pointer
might also not be in the list anymore after __detach_device
returns. So just repeatedly remove the first element of the
list until it is empty.

Cc: stable@vger.kernel.org
Tested-by: NMarti Raudsepp <marti@juffo.org>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

This structure is read-only data and should never be modified.
Signed-off-by: NThierry Reding <treding@nvidia.com>
Signed-off-by: NGreg Kroah-Hartman <gregkh@linuxfoundation.org>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>