Add two new functions to the IOMMU-API to allow the IOMMU
drivers to export the requirements for direct mapped regions
per device.
This is useful for exporting the information in Intel VT-d's
RMRR entries or AMD-Vi's unity mappings.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

This function can be used to request the current domain a
device is attached to.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Make use of the default domain and re-attach a device to it
when it is detached from another domain. Also enforce that a
device has to be in the default domain before it can be
attached to a different domain.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

This patch changes the behavior of the iommu_attach_device
and iommu_detach_device functions. With this change these
functions only work on devices that have their own group.
For all other devices the iommu_group_attach/detach
functions must be used.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

The default domain will be used (if supported by the iommu
driver) when the devices in the iommu group are not attached
to any other domain.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Do not remove the device from the IOMMU while the driver is
still attached.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Make sure we call the ->remove_device call-back on all
devices already initialized with ->add_device when the bus
initialization fails.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Make sure any errors reported from the IOMMU drivers get
progapated back to the IOMMU core.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Write a message to the kernel log when a device is added or
removed from a group and add debug messages to group
allocation and release routines.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Including the function name is only useful for debugging
messages. They don't belong into other messages from the
iommu core.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Stage 1 translation is controlled by two sets of page tables (TTBR0 and
TTBR1) which grow up and down from zero respectively in the ARMv8
translation regime. For the SMMU, we only care about TTBR0 and, in the
case of a 48-bit virtual space, we expect to map virtual addresses 0x0
through to 0xffff_ffff_ffff.

Given that some masters may be incapable of emitting virtual addresses
targetting TTBR1 (e.g. because they sit on a 48-bit bus), the SMMU
architecture allows bit 47 to be sign-extended, halving the virtual
range of TTBR0 but allowing TTBR1 to be used. This is controlled by the
SEP field in TTBCR2.

The SMMU driver incorrectly enables this sign-extension feature, which
causes problems when userspace addresses are programmed into a master
device with the SMMU expecting to map the incoming transactions via
TTBR0; if the top bit of address is set, we will instead get a
translation fault since TTBR1 walks are disabled in the TTBCR.

This patch fixes the issue by disabling sign-extension of a fixed
virtual address bit and instead basing the behaviour on the upstream bus
size: the incoming address is zero extended unless the upstream bus is
only 49 bits wide, in which case bit 48 is used as the sign bit and is
replicated to the upper bits.

Cc: <stable@vger.kernel.org> # v4.0+
Reported-by: NVarun Sethi <varun.sethi@freescale.com>
Signed-off-by: NWill Deacon <will.deacon@arm.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

The rockchip iommu driver references its of_device_id table
from the init function, which fails to build when the table
is undefined:

iommu/rockchip-iommu.c: In function 'rk_iommu_init':
iommu/rockchip-iommu.c:1029:35: error: 'rk_iommu_dt_ids' undeclared (first use in this function)
  np = of_find_matching_node(NULL, rk_iommu_dt_ids);

This removes the #ifdef and the corresponding of_match_ptr wrapper
to make it build both with CONFIG_OF enabled or disabled.
Signed-off-by: NArnd Bergmann <arnd@arndb.de>
Fixes: 425061b0 ("iommu/rockchip: Play nice in multi-platform builds")
Reviewed-by: NThierry Reding <treding@nvidia.com>
Reviewed-by: NHeiko Stuebner <heiko@sntech.de>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

This patch fixes a bug in put_pasid_state_wait that appeared in kernel 4.0
The bug is that pasid_state->count wasn't decremented before entering the
wait_event. Thus, the condition in wait_event will never be true.

The fix is to decrement (atomically) the pasid_state->count before the
wait_event.
Signed-off-by: NOded Gabbay <oded.gabbay@amd.com>
Cc: stable@vger.kernel.org #v4.0
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

When a default page-size for given level should be mapped,
the level encoding must be 0 rather than 7. This fixes an
issue seen on IOMMUv2 hardware, where this encoding is
enforced.
Tested-by: NSuravee Suthikulpanit <Suravee.Suthikulpanit@amd.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Now that fetch_pte returns the page-size of the pte, this
function can be optimized too.
Tested-by: NSuravee Suthikulpanit <Suravee.Suthikulpanit@amd.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Now that fetch_pte returns the page-size of the pte, the
call in this function can also be optimized a little bit.
Tested-by: NSuravee Suthikulpanit <Suravee.Suthikulpanit@amd.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Now that fetch_pte returns the page-size of the pte, this
function can be optimized a lot.
Tested-by: NSuravee Suthikulpanit <Suravee.Suthikulpanit@amd.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Extend the fetch_pte function to also return the page-size
that is mapped by the returned pte.
Tested-by: NSuravee Suthikulpanit <Suravee.Suthikulpanit@amd.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Add code to allocate memory from the contiguous memory
allocator to support coherent allocations larger than 8MB.
Tested-by: NSuravee Suthikulpanit <Suravee.Suthikulpanit@amd.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Don't explicitly add __GFP_ZERO to the allocator flags.
Leave this up to the caller.
Tested-by: NSuravee Suthikulpanit <Suravee.Suthikulpanit@amd.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Detaching a device from its domain at this event is
problematic for several reasons:

	* The device might me in an alias group and
	  detaching it will also detach all other devices in
	  the group. This removes valid DMA mappings from
	  the other devices causing io-page-faults and lets
	  these devices fail.

	* Devices might have unity mappings specified by the
	  IVRS table. These mappings are required for the
	  device even when no device driver is attached.
	  Detaching the device from its domain in driver
	  unbind will also remove these unity mappings.

This patch removes the handling of the BUS_NOTIFY_UNBOUND_DRIVER
event to prevent these issues and align it better with the
behavior of the VT-d driver.
Tested-by: NSuravee Suthikulpanit <Suravee.Suthikulpanit@amd.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Use the new device-notifier event instead of the old
BUS_NOTIFY_DEL_DEVICE to make sure the device driver had a
chance to uninit the device before all its mappings are
teared down.
Tested-by: NSuravee Suthikulpanit <Suravee.Suthikulpanit@amd.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

The SMMU on Tegra30 and Tegra114 supports addressing up to 4 GiB of
physical memory. On Tegra124 the addressable physical memory was
extended to 16 GiB. The page frame number stored in PTEs therefore
requires 20 or 22 bits, depending on SoC generation.

In order to cope with this, compute the proper value at runtime.
Reported-by: NJoseph Lo <josephl@nvidia.com>
Cc: Hiroshi Doyu <hdoyu@nvidia.com>
Signed-off-by: NThierry Reding <treding@nvidia.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

The aperture of the domain should always be available, otherwise drivers
need to attach first before they can use the aperture geometry.

Cc: Hiroshi Doyu <hdoyu@nvidia.com>
Signed-off-by: NThierry Reding <treding@nvidia.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Each address space in the Tegra SMMU provides 4 GiB worth of addresses.

Cc: Hiroshi Doyu <hdoyu@nvidia.com>
Signed-off-by: NThierry Reding <treding@nvidia.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

All drivers have been converted to the new domain_alloc and
domain_free iommu-ops. So remove the old ones and get rid of
iommu_domain->priv too, as this is no longer needed when the
struct iommu_domain is embedded in the private structures of
the iommu drivers.
Tested-by: NThierry Reding <treding@nvidia.com>
Tested-by: NHeiko Stuebner <heiko@sntech.de>
Reviewed-by: NAlex Williamson <alex.williamson@redhat.com>
Acked-by: NWill Deacon <will.deacon@arm.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Implement domain_alloc and domain_free iommu-ops as a
replacement for domain_init/domain_destroy.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Implement domain_alloc and domain_free iommu-ops as a
replacement for domain_init/domain_destroy.
Tested-by: NHeiko Stuebner <heiko@sntech.de>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Implement domain_alloc and domain_free iommu-ops as a
replacement for domain_init/domain_destroy.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Implement domain_alloc and domain_free iommu-ops as a
replacement for domain_init/domain_destroy.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Implement domain_alloc and domain_free iommu-ops as a
replacement for domain_init/domain_destroy.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Implement domain_alloc and domain_free iommu-ops as a
replacement for domain_init/domain_destroy.
Tested-by: NThierry Reding <treding@nvidia.com>
Acked-by: NThierry Reding <treding@nvidia.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Implement domain_alloc and domain_free iommu-ops as a
replacement for domain_init/domain_destroy.
Tested-by: NThierry Reding <treding@nvidia.com>
Acked-by: NThierry Reding <treding@nvidia.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Implement domain_alloc and domain_free iommu-ops as a
replacement for domain_init/domain_destroy.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Implement domain_alloc and domain_free iommu-ops as a
replacement for domain_init/domain_destroy.
Acked-by: NWill Deacon <will.deacon@arm.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Implement the new domain_alloc and domain_free call-backs
and remove the old domain_init/destroy ones.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Get rid of domain_init and domain_destroy and implement
domain_alloc/domain_free instead.
Reviewed-by: NAlex Williamson <alex.williamson@redhat.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Implement the new iommu-ops function pointers and remove the
obsolete domain_init and domain_destroy functions.
Reviewed-by: NAlex Williamson <alex.williamson@redhat.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Check for the new __IOMMU_DOMAIN_PAGING flag before calling
into the iommu drivers ->map and ->unmap call-backs.
Tested-by: NThierry Reding <treding@nvidia.com>
Tested-by: NHeiko Stuebner <heiko@sntech.de>
Reviewed-by: NAlex Williamson <alex.williamson@redhat.com>
Acked-by: NWill Deacon <will.deacon@arm.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

This allows to handle domains differently based on their
type in the future. An IOMMU driver can implement certain
optimizations for DMA-API domains for example.

The domain types can be extended later and some of the
existing domain attributes can be migrated to become domain
flags.
Tested-by: NThierry Reding <treding@nvidia.com>
Tested-by: NHeiko Stuebner <heiko@sntech.de>
Reviewed-by: NAlex Williamson <alex.williamson@redhat.com>
Acked-by: NWill Deacon <will.deacon@arm.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>