Now that no more drivers rely on arbitrary early initialisation via an
of_iommu_init_fn hook, let's clean up the redundant remnants. The
IOMMU_OF_DECLARE() macro needs to remain for now, as the probe-deferral
mechanism has no other nice way to detect built-in drivers before they
have registered themselves, such that it can make the right decision.
Reviewed-by: NSricharan R <sricharan@codeaurora.org>
Signed-off-by: NRobin Murphy <robin.murphy@arm.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Having of_iommu_init() call ipmmu_init() via ipmmu_vmsa_iommu_of_setup()
does nothing that the subsys_initcall wouldn't do slightly later anyway,
since probe-deferral of masters means it is no longer critical to
register the driver super-early. Clean it up.
Signed-off-by: NRobin Murphy <robin.murphy@arm.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Tie in r8a7795 features and update the IOMMU_OF_DECLARE
compat string to include the updated compat string.
Signed-off-by: NMagnus Damm <damm+renesas@opensource.se>
Signed-off-by: NAlex Williamson <alex.williamson@redhat.com>

Introduce support for two bit SL0 bitfield in IMTTBCR
by using a separate feature flag.
Signed-off-by: NMagnus Damm <damm+renesas@opensource.se>
Signed-off-by: NAlex Williamson <alex.williamson@redhat.com>

Introduce a feature to allow opt-out of setting up
IMBUSCR. The default case is unchanged.
Signed-off-by: NMagnus Damm <damm+renesas@opensource.se>
Signed-off-by: NAlex Williamson <alex.williamson@redhat.com>

Write IMCTR both in the root device and the leaf node.

To allow access of IMCTR introduce the following function:
 - ipmmu_ctx_write_all()

While at it also rename context functions:
 - ipmmu_ctx_read() -> ipmmu_ctx_read_root()
 - ipmmu_ctx_write() -> ipmmu_ctx_write_root()
Signed-off-by: NMagnus Damm <damm+renesas@opensource.se>
Signed-off-by: NAlex Williamson <alex.williamson@redhat.com>

The r8a7795 IPMMU supports 40-bit bus mastering. Both
the coherent DMA mask and the streaming DMA mask are
set to unlock the 40-bit address space for coherent
allocations and streaming operations.
Signed-off-by: NMagnus Damm <damm+renesas@opensource.se>
Signed-off-by: NAlex Williamson <alex.williamson@redhat.com>

Hook up IOMMU_OF_DECLARE() support in case CONFIG_IOMMU_DMA
is enabled. The only current supported case for 32-bit ARM
is disabled, however for 64-bit ARM usage of OF is required.
Signed-off-by: NMagnus Damm <damm+renesas@opensource.se>
Signed-off-by: NAlex Williamson <alex.williamson@redhat.com>

Add support for up to 8 contexts. Each context is mapped to one
domain. One domain is assigned one or more slave devices. Contexts
are allocated dynamically and slave devices are grouped together
based on which IPMMU device they are connected to. This makes slave
devices tied to the same IPMMU device share the same IOVA space.
Signed-off-by: NMagnus Damm <damm+renesas@opensource.se>
Signed-off-by: NAlex Williamson <alex.williamson@redhat.com>

Add root device handling to the IPMMU driver by allowing certain
DT compat strings to enable has_cache_leaf_nodes that in turn will
support both root devices with interrupts and leaf devices that
face the actual IPMMU consumer devices.
Signed-off-by: NMagnus Damm <damm+renesas@opensource.se>
Signed-off-by: NAlex Williamson <alex.williamson@redhat.com>

Introduce struct ipmmu_features to track various hardware
and software implementation changes inside the driver for
different kinds of IPMMU hardware. Add use_ns_alias_offset
as a first example of a feature to control if the secure
register bank offset should be used or not.
Signed-off-by: NMagnus Damm <damm+renesas@opensource.se>
Signed-off-by: NAlex Williamson <alex.williamson@redhat.com>

The remaining difference between the ARM-specific and iommu-dma ops is
in the {add,remove}_device implementations, but even those have some
overlap and duplication. By stubbing out the few arm_iommu_*() calls,
we can get rid of the rest of the inline #ifdeffery to both simplify the
code and improve build coverage.
Signed-off-by: NRobin Murphy <robin.murphy@arm.com>
Signed-off-by: NAlex Williamson <alex.williamson@redhat.com>

Now that the IPMMU instance pointer is the only thing remaining in the
private data structure, we no longer need the extra level of indirection
and can simply stash that directlty in the fwspec.
Signed-off-by: NRobin Murphy <robin.murphy@arm.com>
Signed-off-by: NAlex Williamson <alex.williamson@redhat.com>

We go through quite the merry dance in order to find masters behind the
same IPMMU instance, so that we can ensure they are grouped together.
None of which is really necessary, since the master's private data
already points to the particular IPMMU it is associated with, and that
IPMMU instance data is the perfect place to keep track of a per-instance
group directly.
Signed-off-by: NRobin Murphy <robin.murphy@arm.com>
Signed-off-by: NAlex Williamson <alex.williamson@redhat.com>

We have two implementations for ipmmu_ops->alloc depending on
CONFIG_IOMMU_DMA, the difference being whether they accept the
IOMMU_DOMAIN_DMA type or not. However, iommu_dma_get_cookie() is
guaranteed to return an error when !CONFIG_IOMMU_DMA, so if
ipmmu_domain_alloc_dma() was actually checking and handling the return
value correctly, it would behave the same as ipmmu_domain_alloc()
anyway.

Similarly for freeing; iommu_put_dma_cookie() is robust by design.
Signed-off-by: NRobin Murphy <robin.murphy@arm.com>
Signed-off-by: NAlex Williamson <alex.williamson@redhat.com>

In case of error, the function iommu_group_get() returns NULL pointer
not ERR_PTR(). The IS_ERR() test in the return value check should be
replaced with NULL test.

Fixes: 3ae47292 ("iommu/ipmmu-vmsa: Add new IOMMU_DOMAIN_DMA ops")
Signed-off-by: NWei Yongjun <weiyongjun1@huawei.com>
Signed-off-by: NAlex Williamson <alex.williamson@redhat.com>

Now that the core API issues its own post-unmap TLB sync call, push that
operation out from the io-pgtable-arm internals into the users. For now,
we leave the invalidation implicit in the unmap operation, since none of
the current users would benefit much from any change to that.

CC: Magnus Damm <damm+renesas@opensource.se>
CC: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: NRobin Murphy <robin.murphy@arm.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Make these const as they are not modified anywhere.
Signed-off-by: NBhumika Goyal <bhumirks@gmail.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Reserving a free context is both quicker and more likely to fail
(due to limited hardware resources) than setting up a pagetable.
What is more the pagetable init/cleanup code could require
the context to be set up.
Signed-off-by: NOleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>
CC: Robin Murphy <robin.murphy@arm.com>
CC: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Reviewed-by: NRobin Murphy <robin.murphy@arm.com>
Reviewed-by: NLaurent Pinchart <laurent.pinchart@ideasonboard.com>
CC: Joerg Roedel <joro@8bytes.org>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Extend the driver to make use of iommu_device_sysfs_add()/remove()
functions to hook up initial sysfs support.
Suggested-by: NJoerg Roedel <jroedel@suse.de>
Signed-off-by: NMagnus Damm <damm+renesas@opensource.se>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Get rid of now unused device tracking code. Future code should instead
be able to use driver_for_each_device() for this purpose.

This is a simplified version of the following patch from Robin
[PATCH] iommu/ipmmu-vmsa: Clean up group allocation
Signed-off-by: NRobin Murphy <robin.murphy@arm.com>
Signed-off-by: NMagnus Damm <damm+renesas@opensource.se>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Now when both 32-bit and 64-bit code inside the driver is using
fwspec it is possible to replace the utlb handling with fwspec ids
that get populated from ->of_xlate().
Suggested-by: NRobin Murphy <robin.murphy@arm.com>
Signed-off-by: NMagnus Damm <damm+renesas@opensource.se>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Consolidate the 32-bit and 64-bit code to make use of fwspec instead
of archdata for the 32-bit ARM case.

This is a simplified version of the fwspec handling code from Robin
posted as [PATCH] iommu/ipmmu-vmsa: Convert to iommu_fwspec
Signed-off-by: NRobin Murphy <robin.murphy@arm.com>
Signed-off-by: NMagnus Damm <damm+renesas@opensource.se>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

The 32-bit ARM code gets updated to make use of ->of_xlate() and the
code is shared between 64-bit and 32-bit ARM. The of_device_is_available()
check gets dropped since it is included in of_iommu_xlate().
Suggested-by: NRobin Murphy <robin.murphy@arm.com>
Signed-off-by: NMagnus Damm <damm+renesas@opensource.se>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Extend the driver to make use of iommu_device_register()/unregister()
functions together with iommu_device_set_ops() and iommu_set_fwnode().

These used to be part of the earlier posted 64-bit ARM (r8a7795) series but
it turns out that these days they are required on 32-bit ARM as well.
Signed-off-by: NMagnus Damm <damm+renesas@opensource.se>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Fix comma-instead-of-semicolon typo error present
in the latest version of the IPMMU driver.
Signed-off-by: NMagnus Damm <damm+renesas@opensource.se>
Reviewed-by: NGeert Uytterhoeven <geert+renesas@glider.be>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Convert from archdata to iommu_priv via iommu_fwspec on ARM64 but
let 32-bit ARM keep on using archdata for now.

Once the 32-bit ARM code and the IPMMU driver is able to move over
to CONFIG_IOMMU_DMA=y then coverting to fwspec via ->of_xlate() will
be easy.

For now fwspec ids and num_ids are not used to allow code sharing between
32-bit and 64-bit ARM code inside the driver.
Signed-off-by: NMagnus Damm <damm+renesas@opensource.se>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Introduce an alternative set of iommu_ops suitable for 64-bit ARM
as well as 32-bit ARM when CONFIG_IOMMU_DMA=y. Also adjust the
Kconfig to depend on ARM or IOMMU_DMA. Initialize the device
from ->xlate() when CONFIG_IOMMU_DMA=y.
Signed-off-by: NMagnus Damm <damm+renesas@opensource.se>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Break out the domain allocation code into a separate function.

This is preparation for future code sharing.
Signed-off-by: NMagnus Damm <damm+renesas@opensource.se>
Reviewed-by: NJoerg Roedel <jroedel@suse.de>
Reviewed-by: NGeert Uytterhoeven <geert+renesas@glider.be>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Break out the utlb parsing code and dev_data allocation into a
separate function. This is preparation for future code sharing.
Signed-off-by: NMagnus Damm <damm+renesas@opensource.se>
Reviewed-by: NJoerg Roedel <jroedel@suse.de>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Introduce a bitmap for context handing and convert the
interrupt routine to handle all registered contexts.

At this point the number of contexts are still limited.

Also remove the use of the ARM specific mapping variable
from ipmmu_irq() to allow compile on ARM64.
Signed-off-by: NMagnus Damm <damm+renesas@opensource.se>
Reviewed-by: NJoerg Roedel <jroedel@suse.de>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

The IPMMU driver is using DT these days, and platform data is no longer
used by the driver. Remove unused code.
Signed-off-by: NMagnus Damm <damm+renesas@opensource.se>
Reviewed-by: NLaurent Pinchart <laurent.pinchart@ideasonboard.com>
Reviewed-by: NJoerg Roedel <jroedel@suse.de>
Reviewed-by: NGeert Uytterhoeven <geert+renesas@glider.be>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Currently, the IPMMU/VMSA driver supports 32-bit I/O Virtual Addresses
only, and thus sets io_pgtable_cfg.ias = 32.  However, it doesn't force
a 32-bit IOVA space through the IOMMU Domain Geometry.

Hence if a device (e.g. SYS-DMAC) rightfully configures a 40-bit DMA
mask, it will still be handed out a 40-bit IOVA, outside the 32-bit IOVA
space, leading to out-of-bounds accesses of the PGD when mapping the
IOVA.

Force a 32-bit IOMMU Domain Geometry to fix this.
Signed-off-by: NGeert Uytterhoeven <geert+renesas@glider.be>
Reviewed-by: NRobin Murphy <robin.murphy@arm.com>
Reviewed-by: NLaurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Let's fix the error handle of ipmmu_add_device
when failing to find utlbs, otherwise we take a
risk of pontential memleak.
Signed-off-by: NShawn Lin <shawn.lin@rock-chips.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

If CONFIG_PHYS_ADDR_T_64BIT=n:

    drivers/iommu/ipmmu-vmsa.c: In function 'ipmmu_domain_init_context':
    drivers/iommu/ipmmu-vmsa.c:434:2: warning: right shift count >= width of type
      ipmmu_ctx_write(domain, IMTTUBR0, ttbr >> 32);
      ^

As io_pgtable_cfg.arm_lpae_s1_cfg.ttbr[] is an array of u64s, assigning
it to a phys_addr_t may truncates it.  Make ttbr u64 to fix this.
Signed-off-by: NGeert Uytterhoeven <geert+renesas@glider.be>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

IOMMU hardware with range-based TLB maintenance commands can work
happily with the iova and size arguments passed via the tlb_add_flush
callback, but for IOMMUs which require separate commands per entry in
the range, it is not straightforward to infer the necessary granularity
when it comes to issuing the actual commands.

Add an additional argument indicating the granularity for the benefit
of drivers needing to know, and update the ARM LPAE code appropriately
(for non-leaf invalidations we currently just assume the worst-case
page granularity rather than walking the table to check).
Signed-off-by: NRobin Murphy <robin.murphy@arm.com>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

With the correct DMA API calls now integrated into the io-pgtable code,
let that handle the flushing of non-coherent page table updates.
Signed-off-by: NRobin Murphy <robin.murphy@arm.com>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

Implement domain_alloc and domain_free iommu-ops as a
replacement for domain_init/domain_destroy.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

The of_device_id table is supposed to be zero-terminated.
Signed-off-by: NAxel Lin <axel.lin@ingics.com>
Acked-by: NLaurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

When adding a new device the driver loops over all registered IOMMUs and
calls the ipmmu_find_utlbs() function to parse the DT iommus attribute.
The function returns an error when the IOMMU referenced in DT doesn't
match the current IOMMU. The caller incorrectly breaks from the loop
immediately when the error is reported, resulting in only the first
IOMMU being considered.

Fix this, and while at it move code that isn't specific to an IOMMU
instance out of the loop.
Signed-off-by: NLaurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>