Many IOMMUs support multiple page table formats, meaning that any given
domain may only support a subset of the hardware page sizes presented in
iommu_ops->pgsize_bitmap. There are also certain use-cases where the
creator of a domain may want to control which page sizes are used, for
example to force the use of hugepage mappings to reduce pagetable walk
depth.

To this end, add a per-domain pgsize_bitmap to represent the subset of
page sizes actually in use, to make it possible for domains with
different requirements to coexist.
Signed-off-by: NWill Deacon <will.deacon@arm.com>
[rm: hijacked and rebased original patch with new commit message]
Signed-off-by: NRobin Murphy <robin.murphy@arm.com>
Acked-by: NWill Deacon <will.deacon@arm.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

IOMMU drivers that do not support default domains, but make
use of the the group->domain pointer can get that pointer
overwritten with NULL on device add/remove.

Make sure this can't happen by only overwriting the domain
pointer when it is NULL.

Cc: stable@vger.kernel.org # v4.4+
Fixes: 1228236d ('iommu: Move default domain allocation to iommu_group_get_for_dev()')
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Since iommu_map() code added pgsize value to the paddr, trace_map()
used wrong paddr. So, this patch adds "orig_paddr" value in the
iommu_map() to use for the trace_map().
Signed-off-by: NYoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

commit db0fa0cb "scatterlist: use sg_phys()" did replacements of
the form:

    phys_addr_t phys = page_to_phys(sg_page(s));
    phys_addr_t phys = sg_phys(s) & PAGE_MASK;

However, this breaks platforms where sizeof(phys_addr_t) >
sizeof(unsigned long).  Revert for 4.3 and 4.4 to make room for a
combined helper in 4.5.

Cc: <stable@vger.kernel.org>
Cc: Jens Axboe <axboe@fb.com>
Cc: Christoph Hellwig <hch@lst.de>
Cc: Russell King <linux@arm.linux.org.uk>
Cc: David Woodhouse <dwmw2@infradead.org>
Cc: Andrew Morton <akpm@linux-foundation.org>
Fixes: db0fa0cb ("scatterlist: use sg_phys()")
Suggested-by: NJoerg Roedel <joro@8bytes.org>
Reported-by: NVitaly Lavrov <vel21ripn@gmail.com>
Signed-off-by: NDan Williams <dan.j.williams@intel.com>

Now that the iommu core support for iommu groups is not
pci-centric anymore, we can move default domain allocation
to the bus independent iommu_group_get_for_dev() function.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

All callers of iommu_group_get_for_dev() provide a
device_group call-back now, so this fall-back is no longer
needed.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

This function can be used as a device_group call-back and
just allocates one iommu-group per device.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Rename that function to pci_device_group() and export it, so
that IOMMU drivers can use it as their device_group
call-back.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

That call-back is currently unused, change it into a
call-back function for finding the right IOMMU group for a
device.
This is a first step to remove the hard-coded PCI dependency
in the iommu-group code.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Coccinelle cleanup to replace open coded sg to physical address
translations.  This is in preparation for introducing scatterlists that
reference __pfn_t.

// sg_phys.cocci: convert usage page_to_phys(sg_page(sg)) to sg_phys(sg)
// usage: make coccicheck COCCI=sg_phys.cocci MODE=patch

virtual patch

@@
struct scatterlist *sg;
@@

- page_to_phys(sg_page(sg)) + sg->offset
+ sg_phys(sg)

@@
struct scatterlist *sg;
@@

- page_to_phys(sg_page(sg))
+ sg_phys(sg) & PAGE_MASK
Signed-off-by: NDan Williams <dan.j.williams@intel.com>
Signed-off-by: NJens Axboe <axboe@fb.com>

The -ENODEV error just means that the device is not
translated by an IOMMU. We shouldn't bail out of iommu
driver initialization when that happens, as this is a common
scenario on ARM.

Not returning -ENODEV in the drivers would be a bad idea, as
the IOMMU core would have no indication whether a device is
translated or not. This indication is not used at the
moment, but will probably be in the future.

Fixes: 19762d70 ("iommu: Propagate error in add_iommu_group")
Tested-by: NMarek Szyprowski <m.szyprowski@samsung.com>
Tested-by: NEric Auger <eric.auger@linaro.org>
Tested-by: NHeiko Stuebner <heiko@sntech.de>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

The iommu_group_alloc() and iommu_group_get_for_dev()
functions return error pointers, they never return NULL.
Signed-off-by: NDan Carpenter <dan.carpenter@oracle.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

This function can be called by an IOMMU driver to request
that a device's default domain is direct mapped.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

This will be used to handle unity mappings in the iommu
drivers.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Use the information exported by the IOMMU drivers to create
direct mapped regions in the default domains.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Add two new functions to the IOMMU-API to allow the IOMMU
drivers to export the requirements for direct mapped regions
per device.
This is useful for exporting the information in Intel VT-d's
RMRR entries or AMD-Vi's unity mappings.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

This function can be used to request the current domain a
device is attached to.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Make use of the default domain and re-attach a device to it
when it is detached from another domain. Also enforce that a
device has to be in the default domain before it can be
attached to a different domain.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

This patch changes the behavior of the iommu_attach_device
and iommu_detach_device functions. With this change these
functions only work on devices that have their own group.
For all other devices the iommu_group_attach/detach
functions must be used.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

The default domain will be used (if supported by the iommu
driver) when the devices in the iommu group are not attached
to any other domain.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Do not remove the device from the IOMMU while the driver is
still attached.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Make sure we call the ->remove_device call-back on all
devices already initialized with ->add_device when the bus
initialization fails.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Make sure any errors reported from the IOMMU drivers get
progapated back to the IOMMU core.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Write a message to the kernel log when a device is added or
removed from a group and add debug messages to group
allocation and release routines.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Including the function name is only useful for debugging
messages. They don't belong into other messages from the
iommu core.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

iommu_group_alloc might be called very early in case of iommu controllers
activated from of_iommu, so ensure that this part of subsystem is ready
when devices are being populated from device-tree (core_initcall seems to
be okay for this case).
Signed-off-by: NMarek Szyprowski <m.szyprowski@samsung.com>
Tested-by: NJavier Martinez Canillas <javier.martinez@collabora.co.uk>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

All drivers have been converted to the new domain_alloc and
domain_free iommu-ops. So remove the old ones and get rid of
iommu_domain->priv too, as this is no longer needed when the
struct iommu_domain is embedded in the private structures of
the iommu drivers.
Tested-by: NThierry Reding <treding@nvidia.com>
Tested-by: NHeiko Stuebner <heiko@sntech.de>
Reviewed-by: NAlex Williamson <alex.williamson@redhat.com>
Acked-by: NWill Deacon <will.deacon@arm.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Check for the new __IOMMU_DOMAIN_PAGING flag before calling
into the iommu drivers ->map and ->unmap call-backs.
Tested-by: NThierry Reding <treding@nvidia.com>
Tested-by: NHeiko Stuebner <heiko@sntech.de>
Reviewed-by: NAlex Williamson <alex.williamson@redhat.com>
Acked-by: NWill Deacon <will.deacon@arm.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

This allows to handle domains differently based on their
type in the future. An IOMMU driver can implement certain
optimizations for DMA-API domains for example.

The domain types can be extended later and some of the
existing domain attributes can be migrated to become domain
flags.
Tested-by: NThierry Reding <treding@nvidia.com>
Tested-by: NHeiko Stuebner <heiko@sntech.de>
Reviewed-by: NAlex Williamson <alex.williamson@redhat.com>
Acked-by: NWill Deacon <will.deacon@arm.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

These new call-backs defer the allocation and destruction of
'struct iommu_domain' to the iommu driver. This allows
drivers to embed this struct into their private domain
structures and to get rid of the domain_init and
domain_destroy call-backs when all drivers have been
converted.
Tested-by: NThierry Reding <treding@nvidia.com>
Tested-by: NHeiko Stuebner <heiko@sntech.de>
Reviewed-by: NAlex Williamson <alex.williamson@redhat.com>
Acked-by: NWill Deacon <will.deacon@arm.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

The AMD address is dead for a long time already, replace it
with a working one.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

iommu_map() calls trace_map() with iova and size. trace_map()
should report original iova and original size as opposed to
iova and size after they get changed during mapping. size is
always zero at the end of mapping which is useless to report
and iova as it gets incremented, it is not as useful as the
original iova. Change iommu_map() to call trace_map() to
report original iova and original size.
Signed-off-by: NShuah Khan <shuahkh@osg.samsung.com>
Reported-by: NAlex Williamson <alex.williamson@redhat.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Currently map and unmap are implemented as events under a
common trace class declaration. The common class forces
trace_unmap() to require a bogus physical address argument
that it doesn't use. Changing unmap to report unmapped size
will provide useful information for debugging. Remove common
map_unmap trace class and change map and unmap into separate
events as opposed to events under the same class to allow for
differences in the reporting information. In addition, map and
unmap are changed to handle size value as size_t instead of int
to match the passed size value and avoid overflow.
Signed-off-by: NShuah Khan <shuahkh@osg.samsung.com>
Suggested-by: NAlex Williamson <alex.williamson@redhat.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

iommu_unmap() calls trace_unmap() with changed iova and original
size. trace_unmap() should report original iova instead. Change
iommu_unmap() to call trace_unmap() with original iova.
Signed-off-by: NShuah Khan <shuahkh@osg.samsung.com>
Reported-by: NAlex Williamson <alex.williamson@redhat.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

If the IOMMU supports pages smaller than the CPU page size, segments
which lie at offsets within the CPU page may be mapped based on the
finer-grained IOMMU page boundaries. This minimises the amount of
non-buffer memory between the CPU page boundary and the start of the
segment which must be mapped and therefore exposed to the device, and
brings the default iommu_map_sg implementation in line with
iommu_map/unmap with respect to alignment.
Signed-off-by: NRobin Murphy <robin.murphy@arm.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

IOMMU drivers can be initialized from of_iommu helpers. Such drivers don't
need to provide device_add callbacks to operate properly, so there is no
need to fail initialization if the callback is missing.
Acked-by: NArnd Bergmann <arnd@arndb.de>
Signed-off-by: NMarek Szyprowski <m.szyprowski@samsung.com>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

When some part of bus_set_iommu fails it should undo any made changes
and not simply leave everything as is.

This includes unregistering the bus notifier in iommu_bus_init when
add_iommu_group fails and also setting the bus->iommu_ops back to NULL.
Signed-off-by: NHeiko Stuebner <heiko@sntech.de>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

The IOMMU-API works on page boundarys, unlike the DMA-API
which can work with sub-page buffers. The sg->offset
field does not make sense on the IOMMU level, so force it to
be 0. Do some error-path consolidation while at it.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Mapping and unmapping are more often than not in the critical path.
map_sg allows IOMMU driver implementations to optimize the process
of mapping buffers into the IOMMU page tables.

Instead of mapping a buffer one page at a time and requiring potentially
expensive TLB operations for each page, this function allows the driver
to map all pages in one go and defer TLB maintenance until after all
pages have been mapped.

Additionally, the mapping operation would be faster in general since
clients does not have to keep calling map API over and over again for
each physically contiguous chunk of memory that needs to be mapped to a
virtually contiguous region.
Signed-off-by: NOlav Haugan <ohaugan@codeaurora.org>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

iommu_bus_init() registers a bus notifier on the given bus by using
a statically defined notifier block:

  static struct notifier_block iommu_bus_nb = {
          .notifier_call = iommu_bus_notifier,
  };

This same notifier block is used for all busses. This causes a
problem for notifiers registered after iommu has registered this
callback on multiple busses. The problem is that a subsequent
notifier being registered on a bus which has this iommu notifier
will also get linked in to the notifier list of all other busses
which have this iommu notifier.

This patch fixes this by allocating the notifier_block at runtime.
Some error checking is also added to catch any allocation failure
or notifier registration error.
Signed-off-by: NMark Salter <msalter@redhat.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>