Check for the new __IOMMU_DOMAIN_PAGING flag before calling
into the iommu drivers ->map and ->unmap call-backs.
Tested-by: NThierry Reding <treding@nvidia.com>
Tested-by: NHeiko Stuebner <heiko@sntech.de>
Reviewed-by: NAlex Williamson <alex.williamson@redhat.com>
Acked-by: NWill Deacon <will.deacon@arm.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

This allows to handle domains differently based on their
type in the future. An IOMMU driver can implement certain
optimizations for DMA-API domains for example.

The domain types can be extended later and some of the
existing domain attributes can be migrated to become domain
flags.
Tested-by: NThierry Reding <treding@nvidia.com>
Tested-by: NHeiko Stuebner <heiko@sntech.de>
Reviewed-by: NAlex Williamson <alex.williamson@redhat.com>
Acked-by: NWill Deacon <will.deacon@arm.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

These new call-backs defer the allocation and destruction of
'struct iommu_domain' to the iommu driver. This allows
drivers to embed this struct into their private domain
structures and to get rid of the domain_init and
domain_destroy call-backs when all drivers have been
converted.
Tested-by: NThierry Reding <treding@nvidia.com>
Tested-by: NHeiko Stuebner <heiko@sntech.de>
Reviewed-by: NAlex Williamson <alex.williamson@redhat.com>
Acked-by: NWill Deacon <will.deacon@arm.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

The AMD address is dead for a long time already, replace it
with a working one.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

iommu_map() calls trace_map() with iova and size. trace_map()
should report original iova and original size as opposed to
iova and size after they get changed during mapping. size is
always zero at the end of mapping which is useless to report
and iova as it gets incremented, it is not as useful as the
original iova. Change iommu_map() to call trace_map() to
report original iova and original size.
Signed-off-by: NShuah Khan <shuahkh@osg.samsung.com>
Reported-by: NAlex Williamson <alex.williamson@redhat.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Currently map and unmap are implemented as events under a
common trace class declaration. The common class forces
trace_unmap() to require a bogus physical address argument
that it doesn't use. Changing unmap to report unmapped size
will provide useful information for debugging. Remove common
map_unmap trace class and change map and unmap into separate
events as opposed to events under the same class to allow for
differences in the reporting information. In addition, map and
unmap are changed to handle size value as size_t instead of int
to match the passed size value and avoid overflow.
Signed-off-by: NShuah Khan <shuahkh@osg.samsung.com>
Suggested-by: NAlex Williamson <alex.williamson@redhat.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

iommu_unmap() calls trace_unmap() with changed iova and original
size. trace_unmap() should report original iova instead. Change
iommu_unmap() to call trace_unmap() with original iova.
Signed-off-by: NShuah Khan <shuahkh@osg.samsung.com>
Reported-by: NAlex Williamson <alex.williamson@redhat.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

If the IOMMU supports pages smaller than the CPU page size, segments
which lie at offsets within the CPU page may be mapped based on the
finer-grained IOMMU page boundaries. This minimises the amount of
non-buffer memory between the CPU page boundary and the start of the
segment which must be mapped and therefore exposed to the device, and
brings the default iommu_map_sg implementation in line with
iommu_map/unmap with respect to alignment.
Signed-off-by: NRobin Murphy <robin.murphy@arm.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

IOMMU drivers can be initialized from of_iommu helpers. Such drivers don't
need to provide device_add callbacks to operate properly, so there is no
need to fail initialization if the callback is missing.
Acked-by: NArnd Bergmann <arnd@arndb.de>
Signed-off-by: NMarek Szyprowski <m.szyprowski@samsung.com>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

When some part of bus_set_iommu fails it should undo any made changes
and not simply leave everything as is.

This includes unregistering the bus notifier in iommu_bus_init when
add_iommu_group fails and also setting the bus->iommu_ops back to NULL.
Signed-off-by: NHeiko Stuebner <heiko@sntech.de>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

The IOMMU-API works on page boundarys, unlike the DMA-API
which can work with sub-page buffers. The sg->offset
field does not make sense on the IOMMU level, so force it to
be 0. Do some error-path consolidation while at it.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Mapping and unmapping are more often than not in the critical path.
map_sg allows IOMMU driver implementations to optimize the process
of mapping buffers into the IOMMU page tables.

Instead of mapping a buffer one page at a time and requiring potentially
expensive TLB operations for each page, this function allows the driver
to map all pages in one go and defer TLB maintenance until after all
pages have been mapped.

Additionally, the mapping operation would be faster in general since
clients does not have to keep calling map API over and over again for
each physically contiguous chunk of memory that needs to be mapped to a
virtually contiguous region.
Signed-off-by: NOlav Haugan <ohaugan@codeaurora.org>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

iommu_bus_init() registers a bus notifier on the given bus by using
a statically defined notifier block:

  static struct notifier_block iommu_bus_nb = {
          .notifier_call = iommu_bus_notifier,
  };

This same notifier block is used for all busses. This causes a
problem for notifiers registered after iommu has registered this
callback on multiple busses. The problem is that a subsequent
notifier being registered on a bus which has this iommu notifier
will also get linked in to the notifier list of all other busses
which have this iommu notifier.

This patch fixes this by allocating the notifier_block at runtime.
Some error checking is also added to catch any allocation failure
or notifier registration error.
Signed-off-by: NMark Salter <msalter@redhat.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

It turns out that our assumption that aliases are always to the same
slot isn't true.  One particular platform reports an IVRS alias of the
SATA controller (00:11.0) for the legacy IDE controller (00:14.1).
When we hit this, we attempt to use a single IOMMU group for
everything on the same bus, which in this case is the root complex.
We already have multiple groups defined for the root complex by this
point, resulting in multiple WARN_ON hits.

This patch makes these sorts of aliases work again with IOMMU groups
by reworking how we search through the PCI address space to find
existing groups.  This should also now handle looped dependencies and
all sorts of crazy inter-dependencies that we'll likely never see.

The recursion used here should never be very deep.  It's unlikely to
have individual aliases and only theoretical that we'd ever see a
chain where one alias causes us to search through to yet another
alias.  We're also only dealing with PCIe device on a single bus,
which means we'll typically only see multiple slots in use on the root
complex.  Loops are also a theoretically possibility, which I've
tested using fake DMA alias quirks and prevent from causing problems
using a bitmap of the devfn space that's been visited.
Signed-off-by: NAlex Williamson <alex.williamson@redhat.com>
Cc: stable@vger.kernel.org # 3.17
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Signed-off-by: NJoerg Roedel <jroedel@suse.de>

This function will replace the current iommu_domain_has_cap
function and clean up the interface while at it.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Allow compile-time type-checking.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

When a non-PCI device is passed to that function it might
pass group == NULL to iommu_group_add_device() which then
dereferences it and cause a crash this way. Fix it by
just returning an error for non-PCI devices.

Fixes: 104a1c13
Cc: Alex Williamson <alex.williamson@redhat.com>
Acked-by: NAlex Williamson <alex.williamson@redhat.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Check for the ->map and not the ->unmap pointer.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

This structure is read-only data and should never be modified.
Signed-off-by: NThierry Reding <treding@nvidia.com>
Signed-off-by: NGreg Kroah-Hartman <gregkh@linuxfoundation.org>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Currently each IOMMU driver that supports IOMMU groups has its own
code for discovering the base device used in grouping.  This code
is generally not specific to the IOMMU hardware, but to the bus of
the devices managed by the IOMMU.  We can therefore create a common
interface for supporting devices on different buses.
Signed-off-by: NAlex Williamson <alex.williamson@redhat.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Commit 6197ca82 (iommu: Use %pa and %zx instead of casting) introduced the
usage of '%pa', but still kept the '0x', which leads to printing '0x0x'.

Remove the '0x' when '%pa' is used.
Signed-off-by: NFabio Estevam <fabio.estevam@freescale.com>
Signed-off-by: NJoerg Roedel <joro@8bytes.org>

Change iommu driver to call unmap trace event. This iommu_map_unmap class
event can be enabled to trigger when iommu unmap iommu ops is called. Trace
information includes iova, physical address (map event only), and size.

Testing:
Added trace calls to iommu_prepare_identity_map() for testing some of the
conditions that are hard to trigger. Here is the trace from the testing:

       swapper/0-1     [003] ....     1.854102: unmap: IOMMU: iova=0x00000000cb800000 size=0x400
Signed-off-by: NShuah Khan <shuah.kh@samsung.com>
Signed-off-by: NJoerg Roedel <joro@8bytes.org>

Change iommu driver to call map trace event. This iommu_map_unmap class event
can be enabled to trigger when iommu map iommu ops is called. Trace information
includes iova, physical address (map event only), and size.

Testing:
Added trace calls to iommu_prepare_identity_map() for testing some of the
conditions that are hard to trigger. Here is the trace from the testing:

       swapper/0-1     [003] ....     1.854102: map: IOMMU: iova=0x00000000cb800000 paddr=0x00000000cf9fffff size=0x400
Signed-off-by: NShuah Khan <shuah.kh@samsung.com>
Signed-off-by: NJoerg Roedel <joro@8bytes.org>

Change iommu driver to call detach_device_to_domain trace event. This
iommu_device class event can be enabled to trigger when devices are detached
from a domain. Trace information includes device name.

Testing:
Added trace calls to iommu_prepare_identity_map() for testing some of the
conditions that are hard to trigger. Here is the trace from the testing:

       swapper/0-1     [003] ....     1.854102: detach_device_from_domain: IOMMU: device=0000:00:02.0
Signed-off-by: NShuah Khan <shuah.kh@samsung.com>
Signed-off-by: NJoerg Roedel <joro@8bytes.org>

Change iommu driver to call attach_device_to_domain trace event. This
iommu_device class event can be enabled to trigger when devices are attached
to a domain. Trace information includes device name.

Testing:
Added trace calls to iommu_prepare_identity_map() for testing some of the
conditions that are hard to trigger. Here is the trace from the testing:

      swapper/0-1     [003] ....     1.854102: attach_device_to_domain: IOMMU: device=0000:00:02.0
Signed-off-by: NShuah Khan <shuah.kh@samsung.com>
Signed-off-by: NJoerg Roedel <joro@8bytes.org>

Change iommu driver to call remove_device_to_group trace event. This
iommu_group class event can be enabled to trigger when devices get
removed from an iommu group. Trace information includes iommu group id and
device name.

Testing:
Added trace calls to iommu_prepare_identity_map() for testing some of the
conditions that are hard to trigger. Here is the trace from the testing:

       swapper/0-1     [003] ....     1.854101: remove_device_from_group: IOMMU: groupID=0 device=0000:00:02.0
Signed-off-by: NShuah Khan <shuah.kh@samsung.com>
Signed-off-by: NJoerg Roedel <joro@8bytes.org>

Change iommu driver to call add_device_to_group trace event. This iommu_group
class event can be enabled to trigger when devices get added to an iommu group.
Trace information includes iommu group id and device name.

Testing:
The following is trace is generated when intel-iommu driver adds devices to
to iommu groups during boot-time during its initialization:

       swapper/0-1     [003] ....     1.854793: add_device_to_group: IOMMU: groupID=0 device=0000:00:00.0
       swapper/0-1     [003] ....     1.854797: add_device_to_group: IOMMU: groupID=1 device=0000:00:02.0
Signed-off-by: NShuah Khan <shuah.kh@samsung.com>
Signed-off-by: NJoerg Roedel <joro@8bytes.org>

Add tracing feature to iommu to report various iommu events. Classes
iommu_group, iommu_device, and iommu_map_unmap are defined.

iommu_group class events can be enabled to trigger when devices get added
to and removed from an iommu group. Trace information includes iommu group
id and device name.

iommu:add_device_to_group
iommu:remove_device_from_group

iommu_device class events can be enabled to trigger when devices are attached
to and detached from a domain. Trace information includes device name.

iommu:attach_device_to_domain
iommu:detach_device_from_domain

iommu_map_unmap class events can be enabled to trigger when iommu map and
unmap iommu ops. Trace information includes iova, physical address (map event
only), and size.

iommu:map
iommu:unmap
Signed-off-by: NShuah Khan <shuah.kh@samsung.com>
Signed-off-by: NJoerg Roedel <joro@8bytes.org>

printk supports using %pa for phys_addr_t and
%zx for size_t so use those instead of %lx and
casts to unsigned long.

Other miscellaneous changes around this:

Always use 0x%zx for size instead of one use of decimal.
Coalesce format and align arguments.
Signed-off-by: NJoe Perches <joe@perches.com>
Signed-off-by: NJoerg Roedel <joro@8bytes.org>

Signed-off-by: NAlex Williamson <alex.williamson@redhat.com>
Signed-off-by: NJoerg Roedel <joro@8bytes.org>

iommu_map splits requests into pages that the iommu driver reports
that it can handle.  The iommu_unmap path does not do the same.  This
can cause problems not only from callers that might expect the same
behavior as the map path, but even from the failure path of iommu_map,
should it fail at a point where it has mapped and needs to unwind a
set of pages that the iommu driver cannot handle directly.  amd_iommu,
for example, will BUG_ON if asked to unmap a non power of 2 size.

Fix this by extracting and generalizing the sizing code from the
iommu_map path and use it for both map and unmap.
Signed-off-by: NAlex Williamson <alex.williamson@redhat.com>
Signed-off-by: NJoerg Roedel <joro@8bytes.org>

As IOMMU groups are exposed to the user space by their numbers,
the user space can use them in various kernel APIs so the kernel
might need an API to find a group by its ID.

As an example, QEMU VFIO on PPC64 platform needs it to associate
a logical bus number (LIOBN) with a specific IOMMU group in order
to support in-kernel handling of DMA map/unmap requests.

The patch adds the iommu_group_get_by_id(id) function which performs
such search.

v2: fixed reference counting.
Signed-off-by: NAlexey Kardashevskiy <aik@ozlabs.ru>
Acked-by: NAlex Williamson <alex.williamson@redhat.com>
Signed-off-by: NJoerg Roedel <joro@8bytes.org>

Each iommu window can have access permissions associated with it. Extended the
window_enable API to incorporate window access permissions.

In case of PAMU each window can have its specific set of permissions.
Signed-off-by: NVarun Sethi <Varun.Sethi@freescale.com>
Signed-off-by: NJoerg Roedel <joro@8bytes.org>

This is required in case of PAMU, as it can support a window size of up
to 64G (even on 32bit).
Signed-off-by: NVarun Sethi <Varun.Sethi@freescale.com>
Signed-off-by: NJoerg Roedel <joro@8bytes.org>

This attribute can be used to set and get the number of
subwindows on IOMMUs that are window-based.
Signed-off-by: NJoerg Roedel <joro@8bytes.org>

Add the iommu_domain_window_enable() and iommu_domain_window_disable()
functions to the IOMMU-API. These functions will be used to setup
domains that are based on subwindows and not on paging.
Signed-off-by: NJoerg Roedel <joro@8bytes.org>

This attribute of a domain can be queried to find out if the
domain supports setting up page-tables using the iommu_map()
and iommu_unmap() functions.
Signed-off-by: NJoerg Roedel <joro@8bytes.org>

In case the page-size bitmap is zero the code path in
iommu_map and iommu_unmap is undefined. Make it defined and
return -ENODEV in this case.
Signed-off-by: NJoerg Roedel <joro@8bytes.org>

The iommu_init() initializes IOMMU internal structures and data
required for the IOMMU API as iommu_group_alloc().
It is registered as a subsys_initcall now.

One of the IOMMU users is going to be a PCI subsystem on POWER.
It discovers new IOMMU tables during the PCI scan so the logical
place to call iommu_group_alloc() is the moment when a new group
is discovered. However PCI scan is done from subsys_initcall hook
as IOMMU does so PCI hook can be (and is) called before the IOMMU one.

The patch moves IOMMU subsystem initialization one step earlier
to make sure that IOMMU is initialized before PCI scan begins.
Signed-off-by: NAlexey Kardashevskiy <aik@ozlabs.ru>
Signed-off-by: NJoerg Roedel <joro@8bytes.org>