When booted with intel_iommu=ecs_off we were still allocating the PASID
tables even though we couldn't actually use them. We really want to make
the pasid_enabled() macro depend on ecs_enabled().

Which is unfortunate, because currently they're the other way round to
cope with the Broadwell/Skylake problems with ECS.

Instead of having ecs_enabled() depend on pasid_enabled(), which was never
something that made me happy anyway, make it depend in the normal case
on the "broken PASID" bit 28 *not* being set.

Then pasid_enabled() can depend on ecs_enabled() as it should. And we also
don't need to mess with it if we ever see an implementation that has some
features requiring ECS (like PRI) but which *doesn't* have PASID support.
Signed-off-by: NDavid Woodhouse <David.Woodhouse@intel.com>

This will give a little bit of assistance to those developing drivers
using SVM. It might cause a slight annoyance to end-users whose kernel
disables the IOMMU when drivers are trying to use it. But the fix there
is to fix the kernel to enable the IOMMU.
Signed-off-by: NSudeep Dutt <sudeep.dutt@intel.com>
Signed-off-by: NDavid Woodhouse <David.Woodhouse@intel.com>

Largely based on the driver-mode implementation by Jesse Barnes.
Signed-off-by: NDavid Woodhouse <David.Woodhouse@intel.com>

Signed-off-by: NDavid Woodhouse <David.Woodhouse@intel.com>

This provides basic PASID support for endpoint devices, tested with a
version of the i915 driver.
Signed-off-by: NDavid Woodhouse <David.Woodhouse@intel.com>

The behaviour if you enable PASID support after ATS is undefined. So we
have to enable it first, even if we don't know whether we'll need it.

This is safe enough; unless we set up a context that permits it, the device
can't actually *do* anything with it.

Also shift the feature detction to dmar_insert_one_dev_info() as it only
needs to happen once.
Signed-off-by: NDavid Woodhouse <David.Woodhouse@intel.com>

Add CONFIG_INTEL_IOMMU_SVM, and allocate PASID tables on supported hardware.
Signed-off-by: NDavid Woodhouse <David.Woodhouse@intel.com>

As long as we use an identity mapping to work around the worst of the
hardware bugs which caused us to defeature it and change the definition
of the capability bit, we *can* use PASID support on the devices which
advertised it in bit 28 of the Extended Capability Register.

Allow people to do so with 'intel_iommu=pasid28' on the command line.
Signed-off-by: NDavid Woodhouse <David.Woodhouse@intel.com>

The VT-d specification says that "Software must enable ATS on endpoint
devices behind a Root Port only if the Root Port is reported as
supporting ATS transactions."

We walk up the tree to find a Root Port, but for integrated devices we
don't find one — we get to the host bridge. In that case we *should*
allow ATS. Currently we don't, which means that we are incorrectly
failing to use ATS for the integrated graphics. Fix that.

We should never break out of this loop "naturally" with bus==NULL,
since we'll always find bridge==NULL in that case (and now return 1).

So remove the check for (!bridge) after the loop, since it can never
happen. If it did, it would be worthy of a BUG_ON(!bridge). But since
it'll oops anyway in that case, that'll do just as well.

Cc: stable@vger.kernel.org
Signed-off-by: NDavid Woodhouse <David.Woodhouse@intel.com>

In preparation for the installation of a large page, any small page
tables that may still exist in the target IOV address range are
removed.  However, if a scatter/gather list entry is large enough to
fit more than one large page, the address space for any subsequent
large pages is not cleared of conflicting small page tables.

This can cause legitimate mapping requests to fail with errors of the
form below, potentially followed by a series of IOMMU faults:

ERROR: DMA PTE for vPFN 0xfde00 already set (to 7f83a4003 not 7e9e00083)

In this example, a 4MiB scatter/gather list entry resulted in the
successful installation of a large page @ vPFN 0xfdc00, followed by
a failed attempt to install another large page @ vPFN 0xfde00, due to
the presence of a pointer to a small page table @ 0x7f83a4000.

To address this problem, compute the number of large pages that fit
into a given scatter/gather list entry, and use it to derive the
last vPFN covered by the large page(s).

Cc: stable@vger.kernel.org
Signed-off-by: NChristian Zander <christian@nervanasys.com>
Signed-off-by: NDavid Woodhouse <David.Woodhouse@intel.com>

There is a bug in iommu_context_addr() which will always use
the lower context table, even when the upper context table
needs to be used. Fix this issue.

Fixes: 03ecc32c ("iommu/vt-d: support extended root and context entries")
Reported-by: NXiao, Nan <nan.xiao@hp.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Coccinelle cleanup to replace open coded sg to physical address
translations.  This is in preparation for introducing scatterlists that
reference __pfn_t.

// sg_phys.cocci: convert usage page_to_phys(sg_page(sg)) to sg_phys(sg)
// usage: make coccicheck COCCI=sg_phys.cocci MODE=patch

virtual patch

@@
struct scatterlist *sg;
@@

- page_to_phys(sg_page(sg)) + sg->offset
+ sg_phys(sg)

@@
struct scatterlist *sg;
@@

- page_to_phys(sg_page(sg))
+ sg_phys(sg) & PAGE_MASK
Signed-off-by: NDan Williams <dan.j.williams@intel.com>
Signed-off-by: NJens Axboe <axboe@fb.com>

This fixes wrong accesses to iomem introduced by the kdump
fixing code.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

These functions are only used in that file and can be
static.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Found by a coccicheck script.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

When a 'struct device_domain_info' is created as an alias
for another device, this struct will not be re-used when the
real device is encountered. Fix that to avoid duplicate
device_domain_info structures being added.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

For devices without an PCI alias there will be two
device_domain_info structures added. Prevent that by
checking if the alias is different from the device.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

This struct contains all necessary information for the
function already. Also handle the info->dev == NULL case
while at it.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

The code in the locked section does not touch anything
protected by the dmar_global_lock. Remove it from there.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

When this lock is held the device_domain_lock is also
required to make sure the device_domain_info does not vanish
while in use. So this lock can be removed as it gives no
additional protection.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

There is no need to make a difference here between VM and
non-VM domains, so simplify this code here.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Move the code to attach/detach domains to iommus and vice
verce into a single function to make sure there are no
dangling references.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

This makes domain attachment more synchronous with domain
deattachment. The domain<->iommu link is released in
dmar_remove_one_dev_info.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

This allows to do domain->iommu attachment after domain_init
has run.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Rename this function and the ones further down its
call-chain to domain_context_clear_*. In particular this
means:

	iommu_detach_dependent_devices -> domain_context_clear
		   iommu_detach_dev_cb -> domain_context_clear_one_cb
		      iommu_detach_dev -> domain_context_clear_one

These names match a lot better with its
domain_context_mapping counterparts.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Rename the function to dmar_remove_one_dev_info to match is
name better with its dmar_insert_one_dev_info counterpart.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Rename this function to dmar_insert_one_dev_info() to match
the name better with its counter part function
domain_remove_one_dev_info().
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Do the context-mapping of devices from a single place in the
call-path and clean up the other call-sites.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Just call domain_remove_one_dev_info() for all devices in
the domain instead of reimplementing the functionality.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Simplify this function as much as possible with the new
iommu_refcnt field.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

We don't need to do an expensive search for domain-ids
anymore, as we keep track of per-iommu domain-ids.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

This replaces the dmar_domain->iommu_bmp with a similar
reference count array. This allows us to keep track of how
many devices behind each iommu are attached to the domain.

This is necessary for further simplifications and
optimizations to the iommu<->domain attachment code.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

This field is now obsolete because all places use the
per-iommu domain-ids. Kill the remaining uses of this field
and remove it.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

There is no reason for this special handling of the
si_domain. The per-iommu domain-id can be allocated
on-demand like for any other domain. So remove the
pre-allocation code.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

This function can figure out the domain-id to use itself
from the iommu_did array. This is more reliable over
different domain types and brings us one step further to
remove the domain->id field.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Get rid of the special cases for VM domains vs. non-VM
domains and simplify the code further to just handle the
hardware passthrough vs. page-table case.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

There is no reason to pass the translation type through
multiple layers. It can also be determined in the
domain_context_mapping_one function directly.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

The special case for VM domains is not needed, as other
domains could be attached to the iommu in the same way. So
get rid of this special case.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

This array is indexed by the domain-id and contains the
pointers to the domains attached to this iommu. Modern
systems support 65536 domain ids, so that this array has a
size of 512kb, per iommu.

This is a huge waste of space, as the array is usually
sparsely populated. This patch makes the array
two-dimensional and allocates the memory for the domain
pointers on-demand.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

This makes it easier to change the layout of the data
structure later.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>