Add the IOMMU_PRIV attribute, which is used to indicate privileged
mappings.
Reviewed-by: NRobin Murphy <robin.murphy@arm.com>
Tested-by: NRobin Murphy <robin.murphy@arm.com>
Signed-off-by: NMitchel Humpherys <mitchelh@codeaurora.org>
Acked-by: NWill Deacon <will.deacon@arm.com>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

The of_iommu_{set/get}_ops() API is used to associate a device
tree node with a specific set of IOMMU operations. The same
kernel interface is required on systems booting with ACPI, where
devices are not associated with a device tree node, therefore
the interface requires generalization.

The struct device fwnode member represents the fwnode token associated
with the device and the struct it points at is firmware specific;
regardless, it is initialized on both ACPI and DT systems and makes an
ideal candidate to use it to associate a set of IOMMU operations to a
given device, through its struct device.fwnode member pointer, paving
the way for representing per-device iommu_ops (ie an iommu instance
associated with a device).

Convert the DT specific of_iommu_{set/get}_ops() interface to
use struct device.fwnode as a look-up token, making the interface
usable on ACPI systems and rename the data structures and the
registration API so that they are made to represent their usage
more clearly.
Signed-off-by: NLorenzo Pieralisi <lorenzo.pieralisi@arm.com>
Acked-by: NWill Deacon <will.deacon@arm.com>
Reviewed-by: NRobin Murphy <robin.murphy@arm.com>
Reviewed-by: NTomasz Nowicki <tn@semihalf.com>
Tested-by: NHanjun Guo <hanjun.guo@linaro.org>
Tested-by: NTomasz Nowicki <tn@semihalf.com>
Cc: Will Deacon <will.deacon@arm.com>
Cc: Hanjun Guo <hanjun.guo@linaro.org>
Cc: Robin Murphy <robin.murphy@arm.com>
Cc: Joerg Roedel <joro@8bytes.org>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

iommu_group_get_for_dev() expects that the IOMMU driver's device_group
callback return a group with a reference held for the given device.
Whilst allocating a new group is fine, and pci_device_group() correctly
handles reusing an existing group, there is no general means for IOMMU
drivers doing their own group lookup to take additional references on an
existing group pointer without having to also store device pointers or
resort to elaborate trickery.

Add an IOMMU-driver-specific function to fill the hole.
Acked-by: NSricharan R <sricharan@codeaurora.org>
Signed-off-by: NRobin Murphy <robin.murphy@arm.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Introduce a common structure to hold the per-device firmware data that
most IOMMU drivers need to keep track of. This enables us to configure
much of that data from common firmware code, and consolidate a lot of
the equivalent implementations, device look-up tables, etc. which are
currently strewn across IOMMU drivers.

This will also be enable us to address the outstanding "multiple IOMMUs
on the platform bus" problem by tweaking IOMMU API calls to prefer
dev->fwspec->ops before falling back to dev->bus->iommu_ops, and thus
gracefully handle those troublesome systems which we currently cannot.

As the first user, hook up the OF IOMMU configuration mechanism. The
driver-defined nature of DT cells means that we still need the drivers
to translate and add the IDs themselves, but future users such as the
much less free-form ACPI IORT will be much simpler and self-contained.

CC: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Suggested-by: NWill Deacon <will.deacon@arm.com>
Signed-off-by: NRobin Murphy <robin.murphy@arm.com>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

This new call-back will be used by the iommu driver to do
reserve the given dm_region in its iova space before the
mapping is created.

The call-back is temporary until the dma-ops implementation
is part of the common iommu code.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Many IOMMUs support multiple page table formats, meaning that any given
domain may only support a subset of the hardware page sizes presented in
iommu_ops->pgsize_bitmap. There are also certain use-cases where the
creator of a domain may want to control which page sizes are used, for
example to force the use of hugepage mappings to reduce pagetable walk
depth.

To this end, add a per-domain pgsize_bitmap to represent the subset of
page sizes actually in use, to make it possible for domains with
different requirements to coexist.
Signed-off-by: NWill Deacon <will.deacon@arm.com>
[rm: hijacked and rebased original patch with new commit message]
Signed-off-by: NRobin Murphy <robin.murphy@arm.com>
Acked-by: NWill Deacon <will.deacon@arm.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

The priv field from iommu_ops is a hangover from the of_dma_configure
series and isn't actually used. Remove it before it has chance to
spread.
Signed-off-by: NWill Deacon <will.deacon@arm.com>
Acked-by: NLaurent Pinchart <laurent.pinchart@ideasonboard.com>
Acked-by: NWill Deacon <will.deacon@arm.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

On some platforms, MMIO regions might need slightly different treatment
compared to mapping regular memory; add the notion of MMIO mappings to
the IOMMU API's memory type flags, so that callers can let the IOMMU
drivers know to do the right thing.
Signed-off-by: NRobin Murphy <robin.murphy@arm.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

iommu drivers that support the standard DT bindings use a of_xlate
callback pointer, but that is only part of struct iommu_ops when
CONFIG_OF_IOMMU is enabled, leading to build errors in randconfig
builds when that is not provided:

drivers/iommu/mtk_iommu.c:497:2: error: unknown field 'of_xlate' specified in initializer
  .of_xlate = mtk_iommu_of_xlate,
  ^
drivers/iommu/mtk_iommu.c:497:14: error: initialization from incompatible pointer type [-Werror=incompatible-pointer-types]
  .of_xlate = mtk_iommu_of_xlate,
              ^~~~~~~~~~~~~~~~~~
drivers/iommu/mtk_iommu.c:497:14: note: (near initialization for 'mtk_iommu_ops.domain_get_attr')

We can work around it by adding more #ifdefs in each driver, but
it seems nicer to just allow setting the pointer even if it is
unused. This makes the driver code look nicer, and it gives better
compile-time coverage when test building on other architectures.
Signed-off-by: NArnd Bergmann <arnd@arndb.de>
Fixes: 0df4fabe ("iommu/mediatek: Add mt8173 IOMMU driver")
Reviewed-by: NRobin Murphy <robin.murphy@arm.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Update the comments around struct iommu_ops to match
current state and fix a few typos while at it.
Signed-off-by: NMagnus Damm <damm+renesas@opensource.se>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

This function can be used as a device_group call-back and
just allocates one iommu-group per device.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Rename that function to pci_device_group() and export it, so
that IOMMU drivers can use it as their device_group
call-back.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

That call-back is currently unused, change it into a
call-back function for finding the right IOMMU group for a
device.
This is a first step to remove the hard-coded PCI dependency
in the iommu-group code.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Taking inspiration from the existing arch/arm code, break out some
generic functions to interface the DMA-API to the IOMMU-API. This will
do the bulk of the heavy lifting for IOMMU-backed dma-mapping.

Since associating an IOVA allocator with an IOMMU domain is a fairly
common need, rather than introduce yet another private structure just to
do this for ourselves, extend the top-level struct iommu_domain with the
notion. A simple opaque cookie allows reuse by other IOMMU API users
with their various different incompatible allocator types.
Signed-off-by: NRobin Murphy <robin.murphy@arm.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Using __printf attributes helps to detect several format string issues
at compile time (even though -Wformat-security is currently disabled in
Makefile).  For example it can detect when formatting a pointer as a
number, like the issue fixed in commit a3fa71c4 ("wl18xx: show
rx_frames_per_rates as an array as it really is"), or when the arguments
do not match the format string, c.f.  for example commit 5ce1aca8
("reiserfs: fix __RASSERT format string").

To prevent similar bugs in the future, add a __printf attribute to every
function prototype which needs one in include/linux/ and lib/.  These
functions were mostly found by using gcc's -Wsuggest-attribute=format
flag.
Signed-off-by: NNicolas Iooss <nicolas.iooss_linux@m4x.org>
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Cc: Felipe Balbi <balbi@ti.com>
Cc: Joel Becker <jlbec@evilplan.org>
Signed-off-by: NAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>

This function can be called by an IOMMU driver to request
that a device's default domain is direct mapped.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

This will be used to handle unity mappings in the iommu
drivers.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Add two new functions to the IOMMU-API to allow the IOMMU
drivers to export the requirements for direct mapped regions
per device.
This is useful for exporting the information in Intel VT-d's
RMRR entries or AMD-Vi's unity mappings.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

This function can be used to request the current domain a
device is attached to.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

All drivers have been converted to the new domain_alloc and
domain_free iommu-ops. So remove the old ones and get rid of
iommu_domain->priv too, as this is no longer needed when the
struct iommu_domain is embedded in the private structures of
the iommu drivers.
Tested-by: NThierry Reding <treding@nvidia.com>
Tested-by: NHeiko Stuebner <heiko@sntech.de>
Reviewed-by: NAlex Williamson <alex.williamson@redhat.com>
Acked-by: NWill Deacon <will.deacon@arm.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

This allows to handle domains differently based on their
type in the future. An IOMMU driver can implement certain
optimizations for DMA-API domains for example.

The domain types can be extended later and some of the
existing domain attributes can be migrated to become domain
flags.
Tested-by: NThierry Reding <treding@nvidia.com>
Tested-by: NHeiko Stuebner <heiko@sntech.de>
Reviewed-by: NAlex Williamson <alex.williamson@redhat.com>
Acked-by: NWill Deacon <will.deacon@arm.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

These new call-backs defer the allocation and destruction of
'struct iommu_domain' to the iommu driver. This allows
drivers to embed this struct into their private domain
structures and to get rid of the domain_init and
domain_destroy call-backs when all drivers have been
converted.
Tested-by: NThierry Reding <treding@nvidia.com>
Tested-by: NHeiko Stuebner <heiko@sntech.de>
Reviewed-by: NAlex Williamson <alex.williamson@redhat.com>
Acked-by: NWill Deacon <will.deacon@arm.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

This patch adds a new function to the iommu_ops structure to allow an
OF device to be added to a specific IOMMU instance using the recently
merged generic devicetree binding for IOMMUs. The callback (of_xlate)
takes a struct device representing the master and an of_phandle_args
representing the IOMMU and the correspondong IDs for the new master.
Acked-by: NArnd Bergmann <arnd@arndb.de>
Acked-by: NJoerg Roedel <jroedel@suse.de>
Acked-by: NMarek Szyprowski <m.szyprowski@samsung.com>
Tested-by: NRobin Murphy <robin.murphy@arm.com>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

IOMMU drivers must be initialised before any of their upstream devices,
otherwise the relevant iommu_ops won't be configured for the bus in
question. To solve this, a number of IOMMU drivers use initcalls to
initialise the driver before anything has a chance to be probed.

Whilst this solves the immediate problem, it leaves the job of probing
the IOMMU completely separate from the iommu_ops to configure the IOMMU,
which are called on a per-bus basis and require the driver to figure out
exactly which instance of the IOMMU is being requested. In particular,
the add_device callback simply passes a struct device to the driver,
which then has to parse firmware tables or probe buses to identify the
relevant IOMMU instance.

This patch takes the first step in addressing this problem by adding an
early initialisation pass for IOMMU drivers, giving them the ability to
store some per-instance data in their iommu_ops structure and store that
in their of_node. This can later be used when parsing OF masters to
identify the IOMMU instance in question.
Acked-by: NArnd Bergmann <arnd@arndb.de>
Acked-by: NJoerg Roedel <jroedel@suse.de>
Acked-by: NMarek Szyprowski <m.szyprowski@samsung.com>
Tested-by: NRobin Murphy <robin.murphy@arm.com>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

Some IOMMUs accept an IOMMU_NOEXEC protection flag in addition to
IOMMU_READ and IOMMU_WRITE. Expose this as an IOMMU capability.
Signed-off-by: NAntonios Motakis <a.motakis@virtualopensystems.com>
Acked-by: NJoerg Roedel <jroedel@suse.de>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

Exposing the XN flag of the SMMU driver as IOMMU_NOEXEC instead of
IOMMU_EXEC makes it enforceable, since for IOMMUs that don't support
the XN flag pages will always be executable.
Signed-off-by: NAntonios Motakis <a.motakis@virtualopensystems.com>
Acked-by: NJoerg Roedel <jroedel@suse.de>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

Mapping and unmapping are more often than not in the critical path.
map_sg allows IOMMU driver implementations to optimize the process
of mapping buffers into the IOMMU page tables.

Instead of mapping a buffer one page at a time and requiring potentially
expensive TLB operations for each page, this function allows the driver
to map all pages in one go and defer TLB maintenance until after all
pages have been mapped.

Additionally, the mapping operation would be faster in general since
clients does not have to keep calling map API over and over again for
each physically contiguous chunk of memory that needs to be mapped to a
virtually contiguous region.
Signed-off-by: NOlav Haugan <ohaugan@codeaurora.org>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Some IOMMUs, such as the ARM SMMU, support two stages of translation.
The idea behind such a scheme is to allow a guest operating system to
use the IOMMU for DMA mappings in the first stage of translation, with
the hypervisor then installing mappings in the second stage to provide
isolation of the DMA to the physical range assigned to that virtual
machine.

In order to allow IOMMU domains to be used for second-stage translation,
this patch adds a new iommu_attr (IOMMU_ATTR_NESTING) for setting
second-stage domains prior to device attach. The attribute can also be
queried to see if a domain is actually making use of nesting.
Acked-by: NJoerg Roedel <jroedel@suse.de>
Signed-off-by: NWill Deacon <will.deacon@arm.com>
Signed-off-by: NAlex Williamson <alex.williamson@redhat.com>

Signed-off-by: NJoerg Roedel <jroedel@suse.de>

This function will replace the current iommu_domain_has_cap
function and clean up the interface while at it.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Allow compile-time type-checking.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

0-day kernel build testing reports:

   arch/x86/kvm/x86.o: In function `iommu_device_destroy':
>> (.text+0x7a0a): multiple definition of `iommu_device_destroy'
   arch/x86/kvm/../../../virt/kvm/vfio.o:vfio.c:(.text+0x490): first defined here
   arch/x86/kvm/x86.o: In function `iommu_device_link':
>> (.text+0x7a15): multiple definition of `iommu_device_link'
   arch/x86/kvm/../../../virt/kvm/vfio.o:vfio.c:(.text+0x49b): first defined here
   arch/x86/kvm/x86.o: In function `iommu_device_unlink':
>> (.text+0x7a25): multiple definition of `iommu_device_unlink'
   arch/x86/kvm/../../../virt/kvm/vfio.o:vfio.c:(.text+0x4ab): first defined here
   arch/x86/kvm/x86.o: In function `iommu_device_create':
>> (.text+0x79f8): multiple definition of `iommu_device_create'
   arch/x86/kvm/../../../virt/kvm/vfio.o:vfio.c:(.text+0x47e): first defined here

These are due to failing to define the stubs as static inline.  Fix.
Signed-off-by: NAlex Williamson <alex.williamson@redhat.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

This structure is read-only data and should never be modified.
Signed-off-by: NThierry Reding <treding@nvidia.com>
Signed-off-by: NGreg Kroah-Hartman <gregkh@linuxfoundation.org>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

IOMMUs currently have no common representation to userspace, most
seem to have no representation at all aside from a few printks
on bootup.  There are however features of IOMMUs that are useful
to know about.  For instance the IOMMU might support superpages,
making use of processor large/huge pages more important in a device
assignment scenario.  It's also useful to create cross links between
devices and IOMMU hardware units, so that users might be able to
load balance their devices to avoid thrashing a single hardware unit.

This patch adds a device create and destroy interface as well as
device linking, making it very lightweight for an IOMMU driver to add
basic support.  IOMMU drivers can provide additional attributes
automatically by using an attribute_group.

The attributes exposed are expected to be relatively device specific,
the means to retrieve them certainly are, so there are currently no
common attributes for the new class created here.
Signed-off-by: NAlex Williamson <alex.williamson@redhat.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Currently each IOMMU driver that supports IOMMU groups has its own
code for discovering the base device used in grouping.  This code
is generally not specific to the IOMMU hardware, but to the bus of
the devices managed by the IOMMU.  We can therefore create a common
interface for supporting devices on different buses.
Signed-off-by: NAlex Williamson <alex.williamson@redhat.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

domain_has_cap is a misnomer bc the func name should be
the same for CONFIG_IOMMU_API and !CONFIG_IOMMU_API.
Signed-off-by: NUpinder Malhi <umalhi@cisco.com>
Signed-off-by: NJoerg Roedel <joro@8bytes.org>

Almost every function in include/linux/iommu.h has an empty stub
but the iommu_group_get_by_id() did not get one by mistake.

This adds an empty stub for iommu_group_get_by_id() for IOMMU_API
disabled config.
Signed-off-by: NAlexey Kardashevskiy <aik@ozlabs.ru>
Signed-off-by: NJoerg Roedel <joro@8bytes.org>

Whilst most IOMMU mappings should probably be non-executable, there
may be cases (HSA?) where executable mappings are required.

This patch introduces a new mapping flag, IOMMU_EXEC, to indicate that
the mapping should be mapped as executable.
Signed-off-by: NWill Deacon <will.deacon@arm.com>

Change iommu driver call io_page_fault trace event. This iommu_error class
event can be enabled to trigger when an iommu error occurs. Trace information
includes driver name, device name, iova, and flags.

Testing:
Added trace calls to iommu_prepare_identity_map() for testing some of the
conditions that are hard to trigger. Here is the trace from the testing:

       swapper/0-1     [003] ....     2.003774: io_page_fault: IOMMU:pci 0000:00:02.0 iova=0x00000000cb800000 flags=0x0002
       swapper/0-1     [003] ....     2.004098: io_page_fault: IOMMU:pci 0000:00:1d.0 iova=0x00000000cadc6000 flags=0x0002
       swapper/0-1     [003] ....     2.004115: io_page_fault: IOMMU:pci 0000:00:1a.0 iova=0x00000000cadc6000 flags=0x0002
       swapper/0-1     [003] ....     2.004129: io_page_fault: IOMMU:pci 0000:00:1f.0 iova=0x0000000000000000 flags=0x0002
Signed-off-by: NShuah Khan <shuah.kh@samsung.com>
Signed-off-by: NJoerg Roedel <joro@8bytes.org>

Added the following domain attributes for the FSL PAMU driver:
1. Added new iommu stash attribute, which allows setting of the
   LIODN specific stash id parameter through IOMMU API.
2. Added an attribute for enabling/disabling DMA to a particular
   memory window.
3. Added domain attribute to check for PAMUV1 specific constraints.
Signed-off-by: NVarun Sethi <Varun.Sethi@freescale.com>
Signed-off-by: NJoerg Roedel <joro@8bytes.org>