With all IOMMU drivers being converted to bus_set_iommu the
global iommu_ops are no longer required. The same is true
for the deprecated register_iommu function.
Signed-off-by: NJoerg Roedel <joerg.roedel@amd.com>

Convert the MSM IOMMU driver for ARM to use the new
interface for publishing the iommu_ops.
Acked-by: NDavid Brown <davidb@codeaurora.org>
Signed-off-by: NJoerg Roedel <joerg.roedel@amd.com>

Convert the OMAP IOMMU driver on ARM to use the new
interface for publishing the iommu_ops.

Cc: Ohad Ben-Cohen <ohad@wizery.com>
Signed-off-by: NJoerg Roedel <joerg.roedel@amd.com>

Convert the Intel IOMMU driver to use the new interface for
publishing the iommu_ops.

Cc: David Woodhouse <dwmw2@infradead.org>
Signed-off-by: NJoerg Roedel <joerg.roedel@amd.com>

Convert the AMD IOMMU driver to use the new interface for
publishing the iommu_ops.
Signed-off-by: NJoerg Roedel <joerg.roedel@amd.com>

Use the per-bus iommu-ops in the functions of the iommu-api
instead of the global iommu_ops.
Signed-off-by: NJoerg Roedel <joerg.roedel@amd.com>

With per-bus iommu_ops the iommu_found function needs to
work on a bus_type too. This patch adds a bus_type parameter
to that function and converts all call-places.
The function is also renamed to iommu_present because the
function now checks if an iommu is present for a given bus
and does not check for a global iommu anymore.
Signed-off-by: NJoerg Roedel <joerg.roedel@amd.com>

This is necessary to store a pointer to the bus-specific
iommu_ops in the iommu-domain structure. It will be used
later to call into bus-specific iommu-ops.
Signed-off-by: NJoerg Roedel <joerg.roedel@amd.com>

This is the starting point to make the iommu_ops used for
the iommu-api a per-bus-type structure. It is required to
easily implement bus-specific setup in the iommu-layer.
The first user will be the iommu-group attribute in sysfs.
Acked-by: NGreg Kroah-Hartman <gregkh@suse.de>
Signed-off-by: NJoerg Roedel <joerg.roedel@amd.com>

We really don't want this to work in the general case; device drivers
*shouldn't* care whether they are behind an IOMMU or not. But the
integrated graphics is a special case, because the IOMMU and the GTT are
all kind of smashed into one and generally horrifically buggy, so it's
reasonable for the graphics driver to want to know when the IOMMU is
active for the graphics hardware.
Signed-off-by: NDavid Woodhouse <David.Woodhouse@intel.com>
Signed-off-by: NBen Widawsky <ben@bwidawsk.net>
Signed-off-by: NKeith Packard <keithp@keithp.com>

To work around a hardware issue, we have to submit IOTLB flushes while
the graphics engine is idle. The graphics driver will (we hope) go to
great lengths to ensure that it gets that right on the affected
chipset(s)... so let's not screw it over by deferring the unmap and
doing it later. That wouldn't be very helpful.
Signed-off-by: NDavid Woodhouse <David.Woodhouse@intel.com>
Signed-off-by: NBen Widawsky <ben@bwidawsk.net>
Signed-off-by: NKeith Packard <keithp@keithp.com>

If target_level == 0, current code breaks out of the while-loop if
SUPERPAGE bit is set. We should also break out if PTE is not present.
If we don't do this, KVM calls to iommu_iova_to_phys() will cause
pfn_to_dma_pte() to create mapping for 4KiB pages.
Signed-off-by: NAllen Kay <allen.m.kay@intel.com>
Signed-off-by: NDavid Woodhouse <David.Woodhouse@intel.com>

set dmar->iommu_superpage field to the smallest common denominator
of super page sizes supported by all active VT-d engines.  Initialize
this field in intel_iommu_domain_init() API so intel_iommu_map() API
will be able to use iommu_superpage field to determine the appropriate
super page size to use.
Signed-off-by: NAllen Kay <allen.m.kay@intel.com>
Signed-off-by: NDavid Woodhouse <David.Woodhouse@intel.com>

iommu_unmap() API expects IOMMU drivers to return the actual page order
of the address being unmapped.  Previous code was just returning page
order passed in from the caller.  This patch fixes this problem.
Signed-off-by: NAllen Kay <allen.m.kay@intel.com>
Signed-off-by: NDavid Woodhouse <David.Woodhouse@intel.com>

We really don't want this to work in the general case; device drivers
*shouldn't* care whether they are behind an IOMMU or not. But the
integrated graphics is a special case, because the IOMMU and the GTT are
all kind of smashed into one and generally horrifically buggy, so it's
reasonable for the graphics driver to want to know when the IOMMU is
active for the graphics hardware.
Signed-off-by: NDavid Woodhouse <David.Woodhouse@intel.com>

To work around a hardware issue, we have to submit IOTLB flushes while
the graphics engine is idle. The graphics driver will (we hope) go to
great lengths to ensure that it gets that right on the affected
chipset(s)... so let's not screw it over by deferring the unmap and
doing it later. That wouldn't be very helpful.
Signed-off-by: NDavid Woodhouse <David.Woodhouse@intel.com>

The shift direction was wrong because the function takes a
page number and i is the address is the loop.

Cc: stable@kernel.org
Signed-off-by: NJoerg Roedel <joerg.roedel@amd.com>

When unbinding a device so that I could pass it through to a KVM VM, I
got the lockdep report below.  It looks like a legitimate lock
ordering problem:

 - domain_context_mapping_one() takes iommu->lock and calls
   iommu_support_dev_iotlb(), which takes device_domain_lock (inside
   iommu->lock).

 - domain_remove_one_dev_info() starts by taking device_domain_lock
   then takes iommu->lock inside it (near the end of the function).

So this is the classic AB-BA deadlock.  It looks like a safe fix is to
simply release device_domain_lock a bit earlier, since as far as I can
tell, it doesn't protect any of the stuff accessed at the end of
domain_remove_one_dev_info() anyway.

BTW, the use of device_domain_lock looks a bit unsafe to me... it's
at least not obvious to me why we aren't vulnerable to the race below:

  iommu_support_dev_iotlb()
                                          domain_remove_dev_info()

  lock device_domain_lock
    find info
  unlock device_domain_lock

                                          lock device_domain_lock
                                            find same info
                                          unlock device_domain_lock

                                          free_devinfo_mem(info)

  do stuff with info after it's free

However I don't understand the locking here well enough to know if
this is a real problem, let alone what the best fix is.

Anyway here's the full lockdep output that prompted all of this:

     =======================================================
     [ INFO: possible circular locking dependency detected ]
     2.6.39.1+ #1
     -------------------------------------------------------
     bash/13954 is trying to acquire lock:
      (&(&iommu->lock)->rlock){......}, at: [<ffffffff812f6421>] domain_remove_one_dev_info+0x121/0x230

     but task is already holding lock:
      (device_domain_lock){-.-...}, at: [<ffffffff812f6508>] domain_remove_one_dev_info+0x208/0x230

     which lock already depends on the new lock.

     the existing dependency chain (in reverse order) is:

     -> #1 (device_domain_lock){-.-...}:
            [<ffffffff8109ca9d>] lock_acquire+0x9d/0x130
            [<ffffffff81571475>] _raw_spin_lock_irqsave+0x55/0xa0
            [<ffffffff812f8350>] domain_context_mapping_one+0x600/0x750
            [<ffffffff812f84df>] domain_context_mapping+0x3f/0x120
            [<ffffffff812f9175>] iommu_prepare_identity_map+0x1c5/0x1e0
            [<ffffffff81ccf1ca>] intel_iommu_init+0x88e/0xb5e
            [<ffffffff81cab204>] pci_iommu_init+0x16/0x41
            [<ffffffff81002165>] do_one_initcall+0x45/0x190
            [<ffffffff81ca3d3f>] kernel_init+0xe3/0x168
            [<ffffffff8157ac24>] kernel_thread_helper+0x4/0x10

     -> #0 (&(&iommu->lock)->rlock){......}:
            [<ffffffff8109bf3e>] __lock_acquire+0x195e/0x1e10
            [<ffffffff8109ca9d>] lock_acquire+0x9d/0x130
            [<ffffffff81571475>] _raw_spin_lock_irqsave+0x55/0xa0
            [<ffffffff812f6421>] domain_remove_one_dev_info+0x121/0x230
            [<ffffffff812f8b42>] device_notifier+0x72/0x90
            [<ffffffff8157555c>] notifier_call_chain+0x8c/0xc0
            [<ffffffff81089768>] __blocking_notifier_call_chain+0x78/0xb0
            [<ffffffff810897b6>] blocking_notifier_call_chain+0x16/0x20
            [<ffffffff81373a5c>] __device_release_driver+0xbc/0xe0
            [<ffffffff81373ccf>] device_release_driver+0x2f/0x50
            [<ffffffff81372ee3>] driver_unbind+0xa3/0xc0
            [<ffffffff813724ac>] drv_attr_store+0x2c/0x30
            [<ffffffff811e4506>] sysfs_write_file+0xe6/0x170
            [<ffffffff8117569e>] vfs_write+0xce/0x190
            [<ffffffff811759e4>] sys_write+0x54/0xa0
            [<ffffffff81579a82>] system_call_fastpath+0x16/0x1b

     other info that might help us debug this:

     6 locks held by bash/13954:
      #0:  (&buffer->mutex){+.+.+.}, at: [<ffffffff811e4464>] sysfs_write_file+0x44/0x170
      #1:  (s_active#3){++++.+}, at: [<ffffffff811e44ed>] sysfs_write_file+0xcd/0x170
      #2:  (&__lockdep_no_validate__){+.+.+.}, at: [<ffffffff81372edb>] driver_unbind+0x9b/0xc0
      #3:  (&__lockdep_no_validate__){+.+.+.}, at: [<ffffffff81373cc7>] device_release_driver+0x27/0x50
      #4:  (&(&priv->bus_notifier)->rwsem){.+.+.+}, at: [<ffffffff8108974f>] __blocking_notifier_call_chain+0x5f/0xb0
      #5:  (device_domain_lock){-.-...}, at: [<ffffffff812f6508>] domain_remove_one_dev_info+0x208/0x230

     stack backtrace:
     Pid: 13954, comm: bash Not tainted 2.6.39.1+ #1
     Call Trace:
      [<ffffffff810993a7>] print_circular_bug+0xf7/0x100
      [<ffffffff8109bf3e>] __lock_acquire+0x195e/0x1e10
      [<ffffffff810972bd>] ? trace_hardirqs_off+0xd/0x10
      [<ffffffff8109d57d>] ? trace_hardirqs_on_caller+0x13d/0x180
      [<ffffffff8109ca9d>] lock_acquire+0x9d/0x130
      [<ffffffff812f6421>] ? domain_remove_one_dev_info+0x121/0x230
      [<ffffffff81571475>] _raw_spin_lock_irqsave+0x55/0xa0
      [<ffffffff812f6421>] ? domain_remove_one_dev_info+0x121/0x230
      [<ffffffff810972bd>] ? trace_hardirqs_off+0xd/0x10
      [<ffffffff812f6421>] domain_remove_one_dev_info+0x121/0x230
      [<ffffffff812f8b42>] device_notifier+0x72/0x90
      [<ffffffff8157555c>] notifier_call_chain+0x8c/0xc0
      [<ffffffff81089768>] __blocking_notifier_call_chain+0x78/0xb0
      [<ffffffff810897b6>] blocking_notifier_call_chain+0x16/0x20
      [<ffffffff81373a5c>] __device_release_driver+0xbc/0xe0
      [<ffffffff81373ccf>] device_release_driver+0x2f/0x50
      [<ffffffff81372ee3>] driver_unbind+0xa3/0xc0
      [<ffffffff813724ac>] drv_attr_store+0x2c/0x30
      [<ffffffff811e4506>] sysfs_write_file+0xe6/0x170
      [<ffffffff8117569e>] vfs_write+0xce/0x190
      [<ffffffff811759e4>] sys_write+0x54/0xa0
      [<ffffffff81579a82>] system_call_fastpath+0x16/0x1b
Signed-off-by: NRoland Dreier <roland@purestorage.com>
Signed-off-by: NDavid Woodhouse <David.Woodhouse@intel.com>

The iommu module on omap contains a few functions that are
only used by the debug module. These are however only there
when the debug code is built as a module. Since it is possible
to build the debug code into the kernel, the functions should
also be provided for the built-in case.
Signed-off-by: NArnd Bergmann <arnd@arndb.de>
Signed-off-by: NOhad Ben-Cohen <ohad@wizery.com>
Signed-off-by: NJoerg Roedel <joerg.roedel@amd.com>

Make report_iommu_fault() return -ENOSYS whenever an iommu fault
handler isn't installed, so IOMMU drivers can then do their own
platform-specific default behavior if they wanted.

Fault handlers can still return -ENOSYS in case they want to elicit the
default behavior of the IOMMU drivers.
Signed-off-by: NOhad Ben-Cohen <ohad@wizery.com>
Signed-off-by: NJoerg Roedel <joerg.roedel@amd.com>

commit 4f3f8d9d "iommu/core: Add fault reporting mechanism" added
the public iommu_set_fault_handler() symbol but forgot to export it.

Fix that.
Signed-off-by: NOhad Ben-Cohen <ohad@wizery.com>
Signed-off-by: NJoerg Roedel <joerg.roedel@amd.com>

Change the CONFIG_DMAR to CONFIG_INTEL_IOMMU to be consistent
with the other IOMMU options.

Rename the CONFIG_INTR_REMAP to CONFIG_IRQ_REMAP to match the
irq subsystem name.

And define the CONFIG_DMAR_TABLE for the common ACPI DMAR
routines shared by both CONFIG_INTEL_IOMMU and CONFIG_IRQ_REMAP.
Signed-off-by: NSuresh Siddha <suresh.b.siddha@intel.com>
Cc: yinghai@kernel.org
Cc: youquan.song@intel.com
Cc: joerg.roedel@amd.com
Cc: tony.luck@intel.com
Cc: dwmw2@infradead.org
Link: http://lkml.kernel.org/r/20110824001456.558630224@sbsiddha-desk.sc.intel.comSigned-off-by: NIngo Molnar <mingo@elte.hu>

Signed-off-by: NSuresh Siddha <suresh.b.siddha@intel.com>
Cc: yinghai@kernel.org
Cc: youquan.song@intel.com
Cc: joerg.roedel@amd.com
Cc: tony.luck@intel.com
Cc: dwmw2@infradead.org
Link: http://lkml.kernel.org/r/20110824001456.386003047@sbsiddha-desk.sc.intel.comSigned-off-by: NIngo Molnar <mingo@elte.hu>

Before the restructruing of the x86 IOMMU code,
intel_iommu_init() was getting called directly from
pci_iommu_init() and hence needed to explicitly set
dmar_disabled to 1 for the failure conditions of
check_zero_address().

Recent changes don't call intel_iommu_init() if the intel iommu
detection fails as a result of failure in check_zero_address().

So no need for this ifdef and the code inside it.
Signed-off-by: NSuresh Siddha <suresh.b.siddha@intel.com>
Cc: yinghai@kernel.org
Cc: youquan.song@intel.com
Cc: joerg.roedel@amd.com
Cc: tony.luck@intel.com
Cc: dwmw2@infradead.org
Link: http://lkml.kernel.org/r/20110824001456.334878686@sbsiddha-desk.sc.intel.comSigned-off-by: NIngo Molnar <mingo@elte.hu>

Move the IOMMU specific routines to intel-iommu.c leaving the
dmar.c to the common ACPI dmar code shared between DMA-remapping
and Interrupt-remapping.
Signed-off-by: NSuresh Siddha <suresh.b.siddha@intel.com>
Cc: yinghai@kernel.org
Cc: youquan.song@intel.com
Cc: joerg.roedel@amd.com
Cc: tony.luck@intel.com
Cc: dwmw2@infradead.org
Link: http://lkml.kernel.org/r/20110824001456.282401285@sbsiddha-desk.sc.intel.comSigned-off-by: NIngo Molnar <mingo@elte.hu>

Both DMA-remapping aswell as Interrupt-remapping depend on the
dmar dev scope to be initialized. When both DMA and
IRQ-remapping are enabled, we depend on DMA-remapping init code
to call dmar_dev_scope_init(). This resulted in not doing this
init when DMA-remapping was turned off but interrupt-remapping
turned on in the kernel config.

This caused interrupt routing to break with CONFIG_INTR_REMAP=y
and CONFIG_DMAR=n.

This issue was introduced by this commit:

 | commit 9d5ce73a
 | Author: FUJITA Tomonori <fujita.tomonori@lab.ntt.co.jp>
 | Date:   Tue Nov 10 19:46:16 2009 +0900
 |
 |    x86: intel-iommu: Convert detect_intel_iommu to use iommu_init hook

Fix this by calling dmar_dev_scope_init() explicitly from the
interrupt remapping code too.
Reported-by: NAndrew Vasquez <andrew.vasquez@qlogic.com>
Signed-off-by: NSuresh Siddha <suresh.b.siddha@intel.com>
Cc: yinghai@kernel.org
Cc: youquan.song@intel.com
Cc: joerg.roedel@amd.com
Cc: tony.luck@intel.com
Cc: dwmw2@infradead.org
Link: http://lkml.kernel.org/r/20110824001456.229207526@sbsiddha-desk.sc.intel.comSigned-off-by: NIngo Molnar <mingo@elte.hu>

On the platforms which are x2apic and interrupt-remapping
capable, Linux kernel is enabling x2apic even if the BIOS
doesn't. This is to take advantage of the features that x2apic
brings in.

Some of the OEM platforms are running into issues because of
this, as their bios is not x2apic aware. For example, this was
resulting in interrupt migration issues on one of the platforms.
Also if the BIOS SMI handling uses APIC interface to send SMI's,
then the BIOS need to be aware of x2apic mode that OS has
enabled.

On some of these platforms, BIOS doesn't have a HW mechanism to
turnoff the x2apic feature to prevent OS from enabling it.

To resolve this mess, recent changes to the VT-d2 specification:

 http://download.intel.com/technology/computing/vptech/Intel(r)_VT_for_Direct_IO.pdf

includes a mechanism that provides BIOS a way to request system
software to opt out of enabling x2apic mode.

Look at the x2apic optout flag in the DMAR tables before
enabling the x2apic mode in the platform. Also print a warning
that we have disabled x2apic based on the BIOS request.

Kernel boot parameter "intremap=no_x2apic_optout" can be used to
override the BIOS x2apic optout request.
Signed-off-by: NYouquan Song <youquan.song@intel.com>
Signed-off-by: NSuresh Siddha <suresh.b.siddha@intel.com>
Cc: yinghai@kernel.org
Cc: joerg.roedel@amd.com
Cc: tony.luck@intel.com
Cc: dwmw2@infradead.org
Link: http://lkml.kernel.org/r/20110824001456.171766616@sbsiddha-desk.sc.intel.comSigned-off-by: NIngo Molnar <mingo@elte.hu>

Without this patch it is possible to select the VIDEO_OMAP3
driver which then selects OMAP_IOVMM. But the omap iommu
driver is not compiled without IOMMU_SUPPORT enabled. Fix
that by forcing OMAP_IOMMU and OMAP_IOVMM are enabled before
VIDEO_OMAP3 can be selected.

Cc: Ohad Ben-Cohen <ohad@wizery.com>
Cc: iommu@lists.linux-foundation.org
Cc: linux-kernel@vger.kernel.org
Signed-off-by: NJoerg Roedel <joerg.roedel@amd.com>

Start using the generic fault report mechanism, as provided by
the IOMMU core, and remove its now-redundant omap_iommu_set_isr API.

Currently we're only interested in letting upper layers know about the
fault, so in case the faulting device is a remote processor, they could
restart it.

Dynamic PTE/TLB loading is not supported.
Signed-off-by: NOhad Ben-Cohen <ohad@wizery.com>
Signed-off-by: NJoerg Roedel <joerg.roedel@amd.com>

Add iommu fault report mechanism to the IOMMU API, so implementations
could report about mmu faults (translation errors, hardware errors,
etc..).

Fault reports can be used in several ways:
- mere logging
- reset the device that accessed the faulting address (may be necessary
  in case the device is a remote processor for example)
- implement dynamic PTE/TLB loading

A dedicated iommu_set_fault_handler() API has been added to allow
users, who are interested to receive such reports, to provide
their handler.
Signed-off-by: NOhad Ben-Cohen <ohad@wizery.com>
Signed-off-by: NJoerg Roedel <joerg.roedel@amd.com>

Mark this lowlevel IRQ handler as non-threaded. This prevents a boot
crash when "threadirqs" is on the kernel commandline. Also the
interrupt handler is handling hardware critical events which should
not be delayed into a thread.
Signed-off-by: NThomas Gleixner <tglx@linutronix.de>
Cc: stable@kernel.org
Signed-off-by: NIngo Molnar <mingo@kernel.org>

The qi->q_lock lock can be taken in atomic context and therefore
cannot be preempted on -rt - annotate it.

In mainline this change documents the low level nature of
the lock - otherwise there's no functional difference. Lockdep
and Sparse checking will work as usual.
Signed-off-by: NThomas Gleixner <tglx@linutronix.de>
Signed-off-by: NIngo Molnar <mingo@elte.hu>

The irq_2_ir_lock can be taken in atomic context and therefore
cannot be preempted on -rt - annotate it.

In mainline this change documents the low level nature of
the lock - otherwise there's no functional difference. Lockdep
and Sparse checking will work as usual.
Signed-off-by: NThomas Gleixner <tglx@linutronix.de>
Signed-off-by: NIngo Molnar <mingo@elte.hu>

The iommu->register_lock can be taken in atomic context and therefore
must not be preempted on -rt - annotate it.

In mainline this change documents the low level nature of
the lock - otherwise there's no functional difference. Lockdep
and Sparse checking will work as usual.
Signed-off-by: NThomas Gleixner <tglx@linutronix.de>
Signed-off-by: NIngo Molnar <mingo@elte.hu>

Replace the hard-coded 4kb by PAGE_SIZE to make iommu-api
implementations possible on architectures where
PAGE_SIZE != 4kb.
Signed-off-by: NJoerg Roedel <joerg.roedel@amd.com>

Replace iommu's alignment checks with the existing IS_ALIGNED macro,
to drop a few lines of code and utilize IS_ALIGNED's type safety.
Signed-off-by: NOhad Ben-Cohen <ohad@wizery.com>
Signed-off-by: NJoerg Roedel <joerg.roedel@amd.com>

Users of the IOMMU API (kvm specifically) assume that iommu_unmap()
returns the order of the unmapped page (on success).

Fix msm_iommu_unmap() accordingly.
Signed-off-by: NOhad Ben-Cohen <ohad@wizery.com>
Cc: Stepan Moskovchenko <stepanm@codeaurora.org>
Cc: David Brown <davidb@codeaurora.org>
Acked-by: NDavid Brown <davidb@codeaurora.org>
Signed-off-by: NJoerg Roedel <joerg.roedel@amd.com>

Users of the IOMMU API (kvm specifically) assume that iommu_unmap()
returns the order of the unmapped page.

Fix omap_iommu_unmap() to do so and adopt omap-iovmm accordingly.
Signed-off-by: NOhad Ben-Cohen <ohad@wizery.com>
Signed-off-by: NJoerg Roedel <joerg.roedel@amd.com>

Tiny cleanup that removes a redundant 'return' statement.
Signed-off-by: NOhad Ben-Cohen <ohad@wizery.com>
Signed-off-by: NJoerg Roedel <joerg.roedel@amd.com>

omap_iovmm requires page-aligned buffers, and that sometimes causes
omap3isp failures (i.e. whenever the buffer passed from userspace is not
page-aligned).

Remove this limitation by rounding the address of the first page entry
down, and adding the offset back to the device address.
Signed-off-by: NLaurent Pinchart <laurent.pinchart@ideasonboard.com>
Acked-by: NHiroshi DOYU <Hiroshi.DOYU@nokia.com>
[ohad@wizery.com: rebased, but tested only with aligned buffers]
[ohad@wizery.com: slightly edited the commit log]
Signed-off-by: NOhad Ben-Cohen <ohad@wizery.com>
Signed-off-by: NJoerg Roedel <joerg.roedel@amd.com>