Consolidate the 32-bit and 64-bit code to make use of fwspec instead
of archdata for the 32-bit ARM case.

This is a simplified version of the fwspec handling code from Robin
posted as [PATCH] iommu/ipmmu-vmsa: Convert to iommu_fwspec
Signed-off-by: NRobin Murphy <robin.murphy@arm.com>
Signed-off-by: NMagnus Damm <damm+renesas@opensource.se>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

The 32-bit ARM code gets updated to make use of ->of_xlate() and the
code is shared between 64-bit and 32-bit ARM. The of_device_is_available()
check gets dropped since it is included in of_iommu_xlate().
Suggested-by: NRobin Murphy <robin.murphy@arm.com>
Signed-off-by: NMagnus Damm <damm+renesas@opensource.se>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Extend the driver to make use of iommu_device_register()/unregister()
functions together with iommu_device_set_ops() and iommu_set_fwnode().

These used to be part of the earlier posted 64-bit ARM (r8a7795) series but
it turns out that these days they are required on 32-bit ARM as well.
Signed-off-by: NMagnus Damm <damm+renesas@opensource.se>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Fix comma-instead-of-semicolon typo error present
in the latest version of the IPMMU driver.
Signed-off-by: NMagnus Damm <damm+renesas@opensource.se>
Reviewed-by: NGeert Uytterhoeven <geert+renesas@glider.be>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Convert from archdata to iommu_priv via iommu_fwspec on ARM64 but
let 32-bit ARM keep on using archdata for now.

Once the 32-bit ARM code and the IPMMU driver is able to move over
to CONFIG_IOMMU_DMA=y then coverting to fwspec via ->of_xlate() will
be easy.

For now fwspec ids and num_ids are not used to allow code sharing between
32-bit and 64-bit ARM code inside the driver.
Signed-off-by: NMagnus Damm <damm+renesas@opensource.se>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Introduce an alternative set of iommu_ops suitable for 64-bit ARM
as well as 32-bit ARM when CONFIG_IOMMU_DMA=y. Also adjust the
Kconfig to depend on ARM or IOMMU_DMA. Initialize the device
from ->xlate() when CONFIG_IOMMU_DMA=y.
Signed-off-by: NMagnus Damm <damm+renesas@opensource.se>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Break out the domain allocation code into a separate function.

This is preparation for future code sharing.
Signed-off-by: NMagnus Damm <damm+renesas@opensource.se>
Reviewed-by: NJoerg Roedel <jroedel@suse.de>
Reviewed-by: NGeert Uytterhoeven <geert+renesas@glider.be>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Break out the utlb parsing code and dev_data allocation into a
separate function. This is preparation for future code sharing.
Signed-off-by: NMagnus Damm <damm+renesas@opensource.se>
Reviewed-by: NJoerg Roedel <jroedel@suse.de>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Introduce a bitmap for context handing and convert the
interrupt routine to handle all registered contexts.

At this point the number of contexts are still limited.

Also remove the use of the ARM specific mapping variable
from ipmmu_irq() to allow compile on ARM64.
Signed-off-by: NMagnus Damm <damm+renesas@opensource.se>
Reviewed-by: NJoerg Roedel <jroedel@suse.de>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

The IPMMU driver is using DT these days, and platform data is no longer
used by the driver. Remove unused code.
Signed-off-by: NMagnus Damm <damm+renesas@opensource.se>
Reviewed-by: NLaurent Pinchart <laurent.pinchart@ideasonboard.com>
Reviewed-by: NJoerg Roedel <jroedel@suse.de>
Reviewed-by: NGeert Uytterhoeven <geert+renesas@glider.be>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Currently, the IPMMU/VMSA driver supports 32-bit I/O Virtual Addresses
only, and thus sets io_pgtable_cfg.ias = 32.  However, it doesn't force
a 32-bit IOVA space through the IOMMU Domain Geometry.

Hence if a device (e.g. SYS-DMAC) rightfully configures a 40-bit DMA
mask, it will still be handed out a 40-bit IOVA, outside the 32-bit IOVA
space, leading to out-of-bounds accesses of the PGD when mapping the
IOVA.

Force a 32-bit IOMMU Domain Geometry to fix this.
Signed-off-by: NGeert Uytterhoeven <geert+renesas@glider.be>
Reviewed-by: NRobin Murphy <robin.murphy@arm.com>
Reviewed-by: NLaurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Let's fix the error handle of ipmmu_add_device
when failing to find utlbs, otherwise we take a
risk of pontential memleak.
Signed-off-by: NShawn Lin <shawn.lin@rock-chips.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

If CONFIG_PHYS_ADDR_T_64BIT=n:

    drivers/iommu/ipmmu-vmsa.c: In function 'ipmmu_domain_init_context':
    drivers/iommu/ipmmu-vmsa.c:434:2: warning: right shift count >= width of type
      ipmmu_ctx_write(domain, IMTTUBR0, ttbr >> 32);
      ^

As io_pgtable_cfg.arm_lpae_s1_cfg.ttbr[] is an array of u64s, assigning
it to a phys_addr_t may truncates it.  Make ttbr u64 to fix this.
Signed-off-by: NGeert Uytterhoeven <geert+renesas@glider.be>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

IOMMU hardware with range-based TLB maintenance commands can work
happily with the iova and size arguments passed via the tlb_add_flush
callback, but for IOMMUs which require separate commands per entry in
the range, it is not straightforward to infer the necessary granularity
when it comes to issuing the actual commands.

Add an additional argument indicating the granularity for the benefit
of drivers needing to know, and update the ARM LPAE code appropriately
(for non-leaf invalidations we currently just assume the worst-case
page granularity rather than walking the table to check).
Signed-off-by: NRobin Murphy <robin.murphy@arm.com>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

With the correct DMA API calls now integrated into the io-pgtable code,
let that handle the flushing of non-coherent page table updates.
Signed-off-by: NRobin Murphy <robin.murphy@arm.com>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

Implement domain_alloc and domain_free iommu-ops as a
replacement for domain_init/domain_destroy.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

The of_device_id table is supposed to be zero-terminated.
Signed-off-by: NAxel Lin <axel.lin@ingics.com>
Acked-by: NLaurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

When adding a new device the driver loops over all registered IOMMUs and
calls the ipmmu_find_utlbs() function to parse the DT iommus attribute.
The function returns an error when the IOMMU referenced in DT doesn't
match the current IOMMU. The caller incorrectly breaks from the loop
immediately when the error is reported, resulting in only the first
IOMMU being considered.

Fix this, and while at it move code that isn't specific to an IOMMU
instance out of the loop.
Signed-off-by: NLaurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Replace the custom page table allocation implementation with the
standard allocator.

The driver loses the ability to map 64kB chunkgs using the PTE
contiguous hint, hence the removal of the SZ_64K page size from the
IOMMU page sizes bitmap.
Signed-off-by: NLaurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

No board file instantiates the IPMMU using platform data. Now that we
have DT support, get rid of platform data.
Signed-off-by: NLaurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>

Devices such as the system DMA controller are connected to multiple
micro TLBs of the same IOMMU. Support this.

Selective enabling of micro TLBs based on runtime device usage isn't
possible at the moment due to lack of support in the IOMMU and DMA
mapping APIs. Support for devices connected to different IOMMUs is also
unsupported for the same reason.
Signed-off-by: NLaurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>

Make platform data optional when the device is instantiated from DT and
look up the micro-TLB number in the bus master DT node.
Signed-off-by: NLaurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>

The TLB must be invalidated after unmapping memory to remove stale TLB
entries. this was supposed to be performed already, but a bug in the
driver prevented the TLB invalidate function from being called. Fix it.
Signed-off-by: NLaurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>

When clearing PUD or PMD entries the child page table (if any) is freed
and the PUD or PMD entry is then cleared. This result in a small race
condition window during which a free page table could be accessed by the
IPMMU.

Fix it by clearing and flushing the PUD or PMD entry before freeing the
child page table.
Signed-off-by: NLaurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>

The ARM IOMMU mapping needs to be released when attaching the device
fails. Add arm_iommu_release_mapping() to the error code path. This is
safe to call with a NULL mapping, so no specific check is needed.

Cleanup is also missing when failing to create a mapping. Jump to the
error code path in that case instead of returning immediately.
Signed-off-by: NLaurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>

Commit a720b41c ("iommu/arm-smmu: change IOMMU_EXEC to
IOMMU_NOEXEC") has inverted and replaced the IOMMU_EXEC flag with
IOMMU_NOEXEC. Update the driver accordingly.
Signed-off-by: NLaurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Signed-off-by: NAxel Lin <axel.lin@ingics.com>
Acked-by: NLaurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Mapping and unmapping are more often than not in the critical path.
map_sg allows IOMMU driver implementations to optimize the process
of mapping buffers into the IOMMU page tables.

Instead of mapping a buffer one page at a time and requiring potentially
expensive TLB operations for each page, this function allows the driver
to map all pages in one go and defer TLB maintenance until after all
pages have been mapped.

Additionally, the mapping operation would be faster in general since
clients does not have to keep calling map API over and over again for
each physically contiguous chunk of memory that needs to be mapped to a
virtually contiguous region.
Signed-off-by: NOlav Haugan <ohaugan@codeaurora.org>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

A platform_driver does not need to set an owner, it will be populated by the
driver core.
Signed-off-by: NWolfram Sang <wsa@the-dreams.de>

This structure is read-only data and should never be modified.
Signed-off-by: NThierry Reding <treding@nvidia.com>
Signed-off-by: NGreg Kroah-Hartman <gregkh@linuxfoundation.org>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

The function arm_iommu_create_mapping lost the order
parameter. Remove it from this IOMMU driver too to make it
build.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Signed-off-by: NLaurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

We don't support stage 2 translation yet.
Signed-off-by: NLaurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Add support for 2MB block mappings at the PMD level.
Signed-off-by: NLaurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

The IOMMU core will only call us with page sizes advertized as supported
by the driver. We can thus simplify the code by removing loops over PGD
and PMD entries.
Signed-off-by: NLaurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

The driver only supports the 3-level long descriptor format that has no
PUD and always has a PMD.
Signed-off-by: NLaurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

The contiguous hint bit signals to the IOMMU that a range of 16 PTEs
refer to physically contiguous memory. It improves performances by
dividing the number of TLB lookups by 16, effectively implementing 64kB
page sizes.
Signed-off-by: NLaurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

The PTRS_PER_(PUD|PGD|PMD|PTE) macros evaluate to different values
depending on whether LPAE is enabled. The IPMMU driver uses a long
descriptor format regardless of LPAE, making those macros mismatch the
IPMMU configuration on non-LPAE systems.

Replace the macros by driver-specific versions that always evaluate to
the right value.
Signed-off-by: NLaurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

The hardware supports 2MB page sizes, not 1MB.
Signed-off-by: NLaurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Cache the micro-TLB number in archdata allocated in the .add_device
handler instead of looking it up when the deviced is attached and
detached. This simplifies the .attach_dev and .detach_dev operations and
prepares for DT support.
Signed-off-by: NLaurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>