Implement domain_alloc and domain_free iommu-ops as a
replacement for domain_init/domain_destroy.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

The Exynos System MMU driver unconditionally executes code and registers
a struct iommu_ops with the platform bus irrespective of whether it runs
on an Exynos SoC or not. This causes problems in multi-platform kernels
where drivers for other SoCs will no longer be able to register their
own struct iommu_ops or even try to use a struct iommu_ops for an IOMMU
that obviously isn't there.

The smallest fix I could think of is to check for the existence of any
Exynos System MMU devices in the device tree and skip initialization
otherwise.

This fixes a problem on Tegra20 where the DRM driver will try to use the
obviously non-existent Exynos System MMU.
Reported-by: NNicolas Chauvet <kwizart@gmail.com>
Cc: Kukjin Kim <kgene@kernel.org>
Signed-off-by: NThierry Reding <treding@nvidia.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Mapping and unmapping are more often than not in the critical path.
map_sg allows IOMMU driver implementations to optimize the process
of mapping buffers into the IOMMU page tables.

Instead of mapping a buffer one page at a time and requiring potentially
expensive TLB operations for each page, this function allows the driver
to map all pages in one go and defer TLB maintenance until after all
pages have been mapped.

Additionally, the mapping operation would be faster in general since
clients does not have to keep calling map API over and over again for
each physically contiguous chunk of memory that needs to be mapped to a
virtually contiguous region.
Signed-off-by: NOlav Haugan <ohaugan@codeaurora.org>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

A platform_driver does not need to set an owner, it will be populated by the
driver core.
Signed-off-by: NWolfram Sang <wsa@the-dreams.de>

Fixed trivial typos and grammar to improve readability.
Changed w/a to workaround.
Signed-off-by: NSachin Kamat <sachin.kamat@samsung.com>
Acked-by: NRandy Dunlap <rdunlap@infradead.org>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

This structure is read-only data and should never be modified.
Signed-off-by: NThierry Reding <treding@nvidia.com>
Signed-off-by: NGreg Kroah-Hartman <gregkh@linuxfoundation.org>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

So there is no point in checking its return value, which will soon
disappear.
Signed-off-by: NJean Delvare <jdelvare@suse.de>
Signed-off-by: NGreg Kroah-Hartman <gregkh@linuxfoundation.org>

Remove unnecessary "&" from function pointers in exynos_iommu_ops.
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Acked-by: NArnd Bergmann <arnd@arndb.de>

Silences the following type of warnings:
WARNING: Missing a blank line after declarations
Signed-off-by: NSachin Kamat <sachin.kamat@linaro.org>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Remove unnecessary "&" from function pointers in exynos_iommu_ops.
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Acked-by: NArnd Bergmann <arnd@arndb.de>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

This patch contains 2 workaround for the System MMU v3.x.

System MMU v3.2 and v3.3 has FLPD cache that caches first level page
table entries to reduce page table walking latency. However, the
FLPD cache is filled with a first level page table entry even though
it is not accessed by a master H/W because System MMU v3.3
speculatively prefetches page table entries that may be accessed
in the near future by the master H/W.
The prefetched FLPD cache entries are not invalidated by iommu_unmap()
because iommu_unmap() only unmaps and invalidates the page table
entries that is mapped.

Because exynos-iommu driver discards a second level page table when
it needs to be replaced with another second level page table or
a first level page table entry with 1MB mapping, It is required to
invalidate FLPD cache that may contain the first level page table
entry that points to the second level page table.

Another workaround of System MMU v3.3 is initializing the first level
page table entries with the second level page table which is filled
with all zeros. This prevents System MMU prefetches 'fault' first
level page table entry which may lead page fault on access to 16MiB
wide.

System MMU 3.x fetches consecutive page table entries by a page
table walking to maximize bus utilization and to minimize TLB miss
panelty.
Unfortunately, functional problem is raised with the fetching behavior
because it fetches 'fault' page table entries that specifies no
translation information and that a valid translation information will
be written to in the near future. The logic in the System MMU generates
page fault with the cached fault entries that is no longer coherent
with the page table which is updated.

There is another workaround that must be implemented by I/O virtual
memory manager: any two consecutive I/O virtual memory area must have
a hole between the two that is larger than or equal to 128KiB.
Also, next I/O virtual memory area must be started from the next
128KiB boundary.

0            128K           256K               384K             512K
|-------------|---------------|-----------------|----------------|
|area1---------------->|.........hole...........|<--- area2 -----

The constraint is depicted above.
The size is selected by the calculation followed:
 - System MMU can fetch consecutive 64 page table entries at once
   64 * 4KiB = 256KiB. This is the size between 128K ~ 384K of the
   above picture. This style of fetching is 'block fetch'. It fetches
   the page table entries predefined consecutive page table entries
   including the entry that is the reason of the page table walking.
 - System MMU can prefetch upto consecutive 32 page table entries.
   This is the size between 256K ~ 384K.
Signed-off-by: NCho KyongHo <pullip.cho@samsung.com>
Signed-off-by: NShaik Ameer Basha <shaik.ameer@samsung.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

This turns on FLPD_CACHE, ACGEN and SYSSEL.

FLPD_CACHE is a cache of 1st level page table entries that contains
the address of a 2nd level page table to reduce latency of page table
walking.

ACGEN is architectural clock gating that gates clocks by System MMU
itself if it is not active. Note that ACGEN is different from clock
gating by the CPU. ACGEN just gates clocks to the internal logic of
System MMU while clock gating by the CPU gates clocks to the System
MMU.

SYSSEL selects System MMU version in some Exynos SoCs. Some Exynos
SoCs have an option to select System MMU versions exclusively because
the SoCs adopts new System MMU version experimentally.

This also always selects LRU as TLB replacement policy. Selecting TLB
replacement policy is deprecated from System MMU 3.2. TLB in System
MMU 3.3 has single TLB replacement policy, LRU. The bit of MMU_CFG
selecting TLB replacement policy is remained as reserved.

QoS value of page table walking is set to 15 (highst value). System
MMU 3.3 can inherit QoS value of page table walking from its master
H/W's transaction. This new feature is enabled by default and QoS
value written to MMU_CFG is ignored.

This patch also adds simplifies the sysmmu version checking by
introducing some macros.
Signed-off-by: NCho KyongHo <pullip.cho@samsung.com>
Signed-off-by: NShaik Ameer Basha <shaik.ameer@samsung.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

This commit adds device tree support for System MMU.

Also, system mmu handling is improved. Previously, an IOMMU domain is
bound to a System MMU which is not correct. This patch binds an IOMMU
domain with the master device of a System MMU.
Signed-off-by: NCho KyongHo <pullip.cho@samsung.com>
Signed-off-by: NShaik Ameer Basha <shaik.ameer@samsung.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Some redundant error message is removed and some error messages
are changed to error level from debug level.
Signed-off-by: NCho KyongHo <pullip.cho@samsung.com>
Signed-off-by: NShaik Ameer Basha <shaik.ameer@samsung.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Patch written by Antonios Motakis <a.motakis@virtualopensystems.com>:

IOMMU groups are expected by certain users of the IOMMU API,
e.g. VFIO. Since each device is behind its own System MMU, we
can allocate a new IOMMU group for each device.
Reviewed-by: NCho KyongHo <pullip.cho@samsung.com>
Signed-off-by: NAntonios Motakis <a.motakis@virtualopensystems.com>
Signed-off-by: NShaik Ameeer Basha <shaik.ameer@samsung.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

This commit introduces sysmmu_pte_t for page table entries and
sysmmu_iova_t vor I/O virtual address that is manipulated by
exynos-iommu driver. The purpose of the typedef is to remove
dependencies to the driver code from the change of CPU architecture
from 32 bit to 64 bit.
Signed-off-by: NCho KyongHo <pullip.cho@samsung.com>
Signed-off-by: NShaik Ameer Basha <shaik.ameer@samsung.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Since acquiring read_lock is not more frequent than write_lock, it is
not beneficial to use rwlock, this commit changes rwlock to spinlock.
Reviewed-by: NGrant Grundler <grundler@chromium.org>
Signed-off-by: NCho KyongHo <pullip.cho@samsung.com>
Signed-off-by: NShaik Ameer Basha <shaik.ameer@samsung.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

This commit removes custom fault handler. The device drivers that
need to register fault handler can register
with iommu_set_fault_handler().

CC: Grant Grundler <grundler@chromium.org>
Signed-off-by: NCho KyongHo <pullip.cho@samsung.com>
Signed-off-by: NShaik Ameer Basha <shaik.ameer@samsung.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

This patch gates clocks of master H/W as well as clocks of System MMU
if master clocks are specified.

Some Exynos SoCs (i.e. GScalers in Exynos5250) have dependencies in
the gating clocks of master H/W and its System MMU. If a H/W is the
case, accessing control registers of System MMU is prohibited unless
both of the gating clocks of System MMU and its master H/W.

CC: Tomasz Figa <t.figa@samsung.com>
Signed-off-by: NCho KyongHo <pullip.cho@samsung.com>
Signed-off-by: NShaik Ameer Basha <shaik.ameer@samsung.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

This patch uses managed device helper functions in the probe().
Signed-off-by: NCho KyongHo <pullip.cho@samsung.com>
Signed-off-by: NShaik Ameer Basha <shaik.ameer@samsung.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

This patch removes dbgname member from sysmmu_drvdata structure.
Kernel message for debugging already has the name of a single
System MMU node. It also removes some compilation warnings.
Signed-off-by: NCho KyongHo <pullip.cho@samsung.com>
Signed-off-by: NShaik Ameer Basha <shaik.ameer@samsung.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Checking if the probing device has a parent device was just to discover
if the probing device is involved in a power domain when the power
domain controlled by Samsung's custom implementation.
Since generic IO power domain is applied, it is required to remove
the condition to see if the probing device has a parent device.
Signed-off-by: NCho KyongHo <pullip.cho@samsung.com>
Signed-off-by: NShaik Ameer Basha <shaik.ameer@samsung.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

This commit adds cache flush for removed small and large page entries
in exynos_iommu_unmap(). Missing cache flush of removed page table
entries can cause missing page fault interrupt when a master IP
accesses an unmapped area.
Reviewed-by: NTomasz Figa <t.figa@samsung.com>
Tested-by: NGrant Grundler <grundler@chromium.org>
Signed-off-by: NCho KyongHo <pullip.cho@samsung.com>
Signed-off-by: NShaik Ameer Basha <shaik.ameer@samsung.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Prefetch buffer is a cache of System MMU 3.x and caches a block of
page table entries to make effect of larger page with small pages.
However, how to control prefetch buffers and the specifications of
prefetch buffers different from minor versions of System MMU v3.
Prefetch buffers must be controled with care because there are some
restrictions in H/W design.

The interface and implementation to initiate prefetch buffers will
be prepared later.
Signed-off-by: NCho KyongHo <pullip.cho@samsung.com>
Signed-off-by: NShaik Ameer Basha <shaik.ameer@samsung.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

L2TLB is 8-way set-associative TLB with 512 entries. The number of
sets is 64.
A single 4KB(small page) translation information is cached
only to a set whose index is the same with the lower 6 bits of the page
frame number.
A single 64KB(large page) translation information can be
cached to any 16 sets whose top two bits of their indices are the same
with the bit [5:4] of the page frame number.
A single 1MB(section) or larger translation information can be cached to
any set in the TLB.

It is required to invalidate entire sets that may cache the target
translation information to guarantee that the L2TLB has no stale data.
Signed-off-by: NCho KyongHo <pullip.cho@samsung.com>
Signed-off-by: NShaik Ameer Basha <shaik.ameer@samsung.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Since kmalloc() does not guarantee that the allignment of 1KiB when it
allocates 1KiB, it is required to allocate lv2 page table from own
slab that guarantees alignment of 1KiB
Signed-off-by: NCho KyongHo <pullip.cho@samsung.com>
Signed-off-by: NShaik Ameer Basha <shaik.ameer@samsung.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

This patch changes not to panic on any error when updating page table.
Instead prints error messages with callstack.
Signed-off-by: NCho KyongHo <pullip.cho@samsung.com>
Signed-off-by: NShaik Ameer Basha <shaik.ameer@samsung.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Commit 25e9d28d (ARM: EXYNOS: remove system mmu initialization from
exynos tree) removed arch/arm/mach-exynos/mach/sysmmu.h header without
removing remaining use of it from exynos-iommu driver, thus causing a
compilation error.

This patch fixes the error by removing respective include line
from exynos-iommu.c.

Use of __pa and __va macro is changed to virt_to_phys and phys_to_virt
which are recommended in driver code. printk formatting of physical
address is also fixed to %pa.

Also System MMU driver is changed to control only a single instance
of System MMU at a time. Since a single instance of System MMU has only
a single clock descriptor for its clock gating, single address range
for control registers, there is no need to obtain two or more clock
descriptors and ioremaped region.

CC: Tomasz Figa <t.figa@samsung.com>
Signed-off-by: NCho KyongHo <pullip.cho@samsung.com>
Signed-off-by: NShaik Ameer Basha <shaik.ameer@samsung.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

exynos_sysmmu_set_prefbuf() is not called any where.
Signed-off-by: NGrant Grundler <grundler@chromium.org>
Reviewed-by: NCho KyongHo <pullip.cho@samsung.com>
Signed-off-by: NJoerg Roedel <joro@8bytes.org>

This is required in case of PAMU, as it can support a window size of up
to 64G (even on 32bit).
Signed-off-by: NVarun Sethi <Varun.Sethi@freescale.com>
Signed-off-by: NJoerg Roedel <joro@8bytes.org>

'exynos_sysmmu_disable' is used only in this file and can be made static.
Signed-off-by: NSachin Kamat <sachin.kamat@linaro.org>
Signed-off-by: NJoerg Roedel <joro@8bytes.org>

Using list_del_init() instead of list_del() + INIT_LIST_HEAD().

spatch with a semantic match is used to found this problem.
(http://coccinelle.lip6.fr/)
Signed-off-by: NWei Yongjun <yongjun_wei@trendmicro.com.cn>
Signed-off-by: NJoerg Roedel <joerg.roedel@amd.com>

Fixes the following build error introduced by commit 3177bb76
("iommu/exynos: Implement DOMAIN_ATTR_GEOMETRY attribute"):

drivers/iommu/exynos-iommu.c: In function ‘exynos_iommu_domain_init’:
drivers/iommu/exynos-iommu.c:735:2: error: ‘dom’ undeclared
(first use in this function)
Signed-off-by: NSachin Kamat <sachin.kamat@linaro.org>
Acked-by: NKukjin Kim <kgene.kim@samsung.com>
Signed-off-by: NJoerg Roedel <joerg.roedel@amd.com>

Correct spelling typo in debug messages and comments
in drivers/iommu.
Signed-off-by: NMasanari Iida <standby24x7@gmail.com>
Signed-off-by: NJiri Kosina <jkosina@suse.cz>

Implement the attribute for the Samsung Exynos IOMMU driver.
Signed-off-by: NJoerg Roedel <joerg.roedel@amd.com>

This is the System MMU driver and IOMMU API implementation for
EXYNOS SoC platforms. EXYNOS platforms has more than 10 System
MMUs dedicated for each multimedia accelerators.

The System MMU driver is already in arc/arm/plat-s5p but it is
moved to drivers/iommu due to Ohad Ben-Cohen gathered IOMMU
drivers there.

Any device driver in EXYNOS platforms that needs to control its
System MMU must call platform_set_sysmmu() to inform System MMU
driver who will control it. platform_set_sysmmu() is defined in
<mach/sysmmu.h>
Signed-off-by: NKyongHo Cho <pullip.cho@samsung.com>
Acked-by: NJoerg Roedel <joerg.roedel@amd.com>
Signed-off-by: NKukjin Kim <kgene.kim@samsung.com>