Currently, system fails to boot because the legacy interrupt remapping
mode does not enable 128-bit IRTE (GA), which is required for x2APIC
support.

Fix by using AMD_IOMMU_GUEST_IR_LEGACY_GA mode when booting with
kernel option amd_iommu_intr=legacy instead. The initialization
logic will check GASup and automatically fallback to using
AMD_IOMMU_GUEST_IR_LEGACY if GA mode is not supported.

Fixes: 3928aa3f ("iommu/amd: Detect and enable guest vAPIC support")
Signed-off-by: NSuravee Suthikulpanit <suravee.suthikulpanit@amd.com>
Link: https://lore.kernel.org/r/1587562202-14183-1-git-send-email-suravee.suthikulpanit@amd.comSigned-off-by: NJoerg Roedel <jroedel@suse.de>

Serious screen flickering when Stoney Ridge outputs to a 4K monitor.

Use identity-mapping and PCI ATS doesn't help this issue.

According to Alex Deucher, IOMMU isn't enabled on Windows, so let's do
the same here to avoid screen flickering on 4K monitor.

Cc: Alex Deucher <alexander.deucher@amd.com>
Bug: https://gitlab.freedesktop.org/drm/amd/issues/961Signed-off-by: NKai-Heng Feng <kai.heng.feng@canonical.com>
Acked-by: NAlex Deucher <alexander.deucher@amd.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

init_iommu_perf_ctr() clobbers the register when it checks write access
to IOMMU perf counters and fails to restore when they are writable.

Add save and restore to fix it.
Signed-off-by: NShuah Khan <skhan@linuxfoundation.org>
Fixes: 30861ddc ("perf/x86/amd: Add IOMMU Performance Counter resource management")
Reviewed-by: NSuravee Suthikulpanit <suravee.suthikulpanit@amd.com>
Tested-by: NSuravee Suthikulpanit <suravee.suthikulpanit@amd.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Optimize the reigster reading by using readq instead of the two
consecutive readl calls.
Signed-off-by: NAdrian Huang <ahuang12@lenovo.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

The bit 13 and bit 14 of the IOMMU control register are
PPRLogEn and PPRIntEn. They are related to PPR (Peripheral Page
Request) instead of 'PPF'. Fix them accrodingly.
Signed-off-by: NAdrian Huang <ahuang12@lenovo.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

The usage of the local variables 'range' and 'misc' was removed
from commit 226e889b ("iommu/amd: Remove first/last_device handling")
and commit 23c742db ("iommu/amd: Split out PCI related parts of
IOMMU initialization"). So, remove them accrodingly.
Signed-off-by: NAdrian Huang <ahuang12@lenovo.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

ioremap has provided non-cached semantics by default since the Linux 2.6
days, so remove the additional ioremap_nocache interface.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Acked-by: NArnd Bergmann <arnd@arndb.de>

The iommu variable in set_device_exclusion_range() us unused
now and causes a compiler warning. Remove it.

Fixes: 387caf0b ("iommu/amd: Treat per-device exclusion ranges as r/w unity-mapped regions")
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Current implementation for IOMMU x2APIC support makes use of
the MMIO access to MSI capability block registers, which requires
checking EFR[MsiCapMmioSup]. However, only IVHD type 11h/40h contain
the information, and not in the IVHD type 10h IOMMU feature reporting
field. Since the BIOS in newer systems, which supports x2APIC, would
normally contain IVHD type 11h/40h, remove the IOMMU_FEAT_XTSUP_SHIFT
check for IVHD type 10h, and only support x2APIC with IVHD type 11h/40h.

Fixes: 66929812 ('iommu/amd: Add support for X2APIC IOMMU interrupts')
Signed-off-by: NSuravee Suthikulpanit <suravee.suthikulpanit@amd.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

The IOMMU MMIO access to MSI capability registers is available only if
the EFR[MsiCapMmioSup] is set. Current implementation assumes this bit
is set if the EFR[XtSup] is set, which might not be the case.

Fix by checking the EFR[MsiCapMmioSup] before accessing the MSI address
low/high and MSI data registers via the MMIO.

Fixes: 66929812 ('iommu/amd: Add support for X2APIC IOMMU interrupts')
Signed-off-by: NSuravee Suthikulpanit <suravee.suthikulpanit@amd.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Some buggy BIOSes might define multiple exclusion ranges of the
IVMD entries which are associated with the same IOMMU hardware.
This leads to the overwritten exclusion range (exclusion_start
and exclusion_length members) in set_device_exclusion_range().

Here is a real case:
When attaching two Broadcom RAID controllers to a server, the first
one reports the failure during booting (the disks connecting to the
RAID controller cannot be detected).

This patch prevents the issue by treating per-device exclusion
ranges as r/w unity-mapped regions.

Discussion:
  * https://lists.linuxfoundation.org/pipermail/iommu/2019-November/040140.htmlSuggested-by: NJoerg Roedel <joro@8bytes.org>
Signed-off-by: NAdrian Huang <ahuang12@lenovo.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Raven Ridge systems may have malfunction touchpad or hang at boot if
incorrect IVRS IOAPIC is provided by BIOS.

Users already found correct "ivrs_ioapic=" values, let's put them inside
kernel to workaround buggy BIOS.

BugLink: https://bugs.launchpad.net/bugs/1795292
BugLink: https://bugs.launchpad.net/bugs/1837688Reported-by: Nkbuild test robot <lkp@intel.com>
Signed-off-by: NKai-Heng Feng <kai.heng.feng@canonical.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

AMD IOMMU requires IntCapXT registers to be setup in order to generate
its own interrupts (for Event Log, PPR Log, and GA Log) with 32-bit
APIC destination ID. Without this support, AMD IOMMU MSI interrupts
will not be routed correctly when booting the system in X2APIC mode.

Cc: Joerg Roedel <joro@8bytes.org>
Fixes: 90fcffd9 ('iommu/amd: Add support for IOMMU XT mode')
Signed-off-by: NSuravee Suthikulpanit <suravee.suthikulpanit@amd.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

When amd_iommu=off was specified on the command line, free_X_resources
functions were called immediately after early_amd_iommu_init. They were
then called again when amd_iommu_init also failed (as expected).

Instead, call them only once: at the end of state_next() whenever
there's an error. These functions should be safe to call any time and
any number of times. However, since state_next is never called again in
an error state, the cleanup will only ever be run once.

This also ensures that cleanup code is run as soon as possible after an
error is detected rather than waiting for amd_iommu_init() to be called.
Signed-off-by: NKevin Mitchell <kevmitch@arista.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

The fallback to the GART driver in the case amd_iommu doesn't work was
executed in a function called free_iommu_resources, which didn't really
make sense. This was even being called twice if amd_iommu=off was
specified on the command line.

The only complication is that it needs to be verified that amd_iommu has
fully relinquished control by calling free_iommu_resources and emptying
the amd_iommu_list.
Signed-off-by: NKevin Mitchell <kevmitch@arista.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Make it safe to call iommu_disable during early init error conditions
before mmio_base is set, but after the struct amd_iommu has been added
to the amd_iommu_list. For example, this happens if firmware fails to
fill in mmio_phys in the ACPI table leading to a NULL pointer
dereference in iommu_feature_disable.

Fixes: 2c0ae172 ('iommu/amd: Convert iommu initialization to state machine')
Signed-off-by: NKevin Mitchell <kevmitch@arista.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Based on 1 normalized pattern(s):

  this program is free software you can redistribute it and or modify
  it under the terms of the gnu general public license version 2 as
  published by the free software foundation this program is
  distributed in the hope that it will be useful but without any
  warranty without even the implied warranty of merchantability or
  fitness for a particular purpose see the gnu general public license
  for more details you should have received a copy of the gnu general
  public license along with this program if not write to the free
  software foundation inc 59 temple place suite 330 boston ma 02111
  1307 usa

extracted by the scancode license scanner the SPDX license identifier

  GPL-2.0-only

has been chosen to replace the boilerplate/reference in 136 file(s).
Signed-off-by: NThomas Gleixner <tglx@linutronix.de>
Reviewed-by: NAlexios Zavras <alexios.zavras@intel.com>
Reviewed-by: NAllison Randal <allison@lohutok.net>
Cc: linux-spdx@vger.kernel.org
Link: https://lkml.kernel.org/r/20190530000436.384967451@linutronix.deSigned-off-by: NGreg Kroah-Hartman <gregkh@linuxfoundation.org>

This variable hold a global list of allocated protection
domains in the AMD IOMMU driver. By now this list is never
traversed anymore, so the list and the lock protecting it
can be removed.

Cc: Tom Murphy <tmurphy@arista.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

The exlcusion range limit register needs to contain the
base-address of the last page that is part of the range, as
bits 0-11 of this register are treated as 0xfff by the
hardware for comparisons.

So correctly set the exclusion range in the hardware to the
last page which is _in_ the range.

Fixes: b2026aa2 ('x86, AMD IOMMU: add functions for programming IOMMU MMIO space')
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

If a device has an exclusion range specified in the IVRS
table, this region needs to be reserved in the iova-domain
of that device. This hasn't happened until now and can cause
data corruption on data transfered with these devices.

Treat exclusion ranges as reserved regions in the iommu-core
to fix the problem.

Fixes: be2a022c ('x86, AMD IOMMU: add functions to parse IOMMU memory mapping requirements for devices')
Signed-off-by: NJoerg Roedel <jroedel@suse.de>
Reviewed-by: NGary R Hook <gary.hook@amd.com>

Use dev_printk() when possible so the IOMMU messages are more consistent
with other messages related to the device.
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Make use of pr_fmt instead of having the 'AMD-Vi' prefix
added manually at every printk() call.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

This register should have been programmed with the physical address
of the memory location containing the shadow tail pointer for
the guest virtual APIC log instead of the base address.

Fixes: 8bda0cfb  ('iommu/amd: Detect and initialize guest vAPIC log')
Signed-off-by: NFilippo Sironi <sironi@amazon.de>
Signed-off-by: NWei Wang <wawei@amazon.de>
Signed-off-by: NSuravee Suthikulpanit <suravee.suthikulpanit@amd.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

The kdump kernel copies the IOMMU device table from the old device table
which is encrypted when SME is enabled in the first kernel. So remap the
old device table with the memory encryption mask in the kdump kernel.

 [ bp: Massage commit message. ]
Signed-off-by: NLianbo Jiang <lijiang@redhat.com>
Signed-off-by: NBorislav Petkov <bp@suse.de>
Reviewed-by: NTom Lendacky <thomas.lendacky@amd.com>
Acked-by: NJoerg Roedel <jroedel@suse.de>
Cc: kexec@lists.infradead.org
Cc: tglx@linutronix.de
Cc: mingo@redhat.com
Cc: hpa@zytor.com
Cc: akpm@linux-foundation.org
Cc: dan.j.williams@intel.com
Cc: bhelgaas@google.com
Cc: baiyaowei@cmss.chinamobile.com
Cc: tiwai@suse.de
Cc: brijesh.singh@amd.com
Cc: dyoung@redhat.com
Cc: bhe@redhat.com
Link: https://lkml.kernel.org/r/20180930031033.22110-4-lijiang@redhat.com

The function is only called from another __init function, so
it should be moved to .init too.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

The AMD IOMMU XT mode enables interrupt remapping with 32-bit destination
APIC ID, which is required for x2APIC. The feature is available when
the XTSup bit is set in the IOMMU Extended Feature register
and/or the IVHD Type 10h IOMMU Feature Reporting field.

For more information, please see section "IOMMU x2APIC Support" of
the AMD I/O Virtualization Technology (IOMMU) Specification.

Cc: Joerg Roedel <jroedel@suse.de>
Signed-off-by: NSuravee Suthikulpanit <suravee.suthikulpanit@amd.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Currently, the driver only supports lower 32-bit of IOMMU Control register.
However, newer AMD IOMMU specification has extended this register
to 64-bit. Therefore, replace the accessing API with the 64-bit version.

Cc: Joerg Roedel <jroedel@suse.de>
Signed-off-by: NSuravee Suthikulpanit <suravee.suthikulpanit@amd.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Implement a skeleton framework for debugfs support in the AMD
IOMMU.  Add an AMD-specific Kconfig boolean that depends upon
general enablement of DebugFS in the IOMMU.
Signed-off-by: NGary R Hook <gary.hook@amd.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Several functions in this driver are called from atomic context,
and thus raw locks must be used in order to be safe on PREEMPT_RT.

This includes paths that must wait for command completion, which is
a potential PREEMPT_RT latency concern but not easily avoidable.
Signed-off-by: NScott Wood <swood@redhat.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

pci_get_bus_and_slot() is restrictive such that it assumes domain=0 as
where a PCI device is present. This restricts the device drivers to be
reused for other domain numbers.

Getting ready to remove pci_get_bus_and_slot() function in favor of
pci_get_domain_bus_and_slot().

Hard-code the domain number as 0 for the AMD IOMMU driver.
Signed-off-by: NSinan Kaya <okaya@codeaurora.org>
Signed-off-by: NBjorn Helgaas <helgaas@kernel.org>
Reviewed-by: NGary R Hook <gary.hook@amd.com>
Acked-by: NJoerg Roedel <jroedel@suse.de>

pr_err() messages should end with a new-line to avoid other messages
being concatenated. So replace '/n' with '\n'.
Signed-off-by: NArvind Yadav <arvind.yadav.cs@gmail.com>
Fixes: 45a01c42 ('iommu/amd: Add function copy_dev_tables()')
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

The variable amd_iommu_pre_enabled is used in non-init
code-paths, so remove the __initdata annotation.
Reported-by: Nkbuild test robot <fengguang.wu@intel.com>
Fixes: 3ac3e5ee ('iommu/amd: Copy old trans table from old kernel')
Acked-by: NBaoquan He <bhe@redhat.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

This was reported by the kbuild bot. The condition in which
entry would be used uninitialized can not happen, because
when there is no iommu this function would never be called.
But its no fast-path, so fix the warning anyway.
Reported-by: Nkbuild test robot <fengguang.wu@intel.com>
Acked-by: NBaoquan He <bhe@redhat.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

It's ok to disable iommu early in normal kernel or in kdump kernel when
amd_iommu=off is specified. While we should not disable it in kdump kernel
when on-flight dma is still on-going.
Signed-off-by: NBaoquan He <bhe@redhat.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

When iommu is pre_enabled in kdump kernel, if a device is set up with
guest translations (DTE.GV=1), then don't copy GCR3 table root pointer
but move the device over to an empty guest-cr3 table and handle the
faults in the PPR log (which answer them with INVALID). After all these
PPR faults are recoverable for the device and we should not allow the
device to change old-kernels data when we don't have to.
Signed-off-by: NBaoquan He <bhe@redhat.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

AMD pointed out it's unsafe to update the device-table while iommu
is enabled. It turns out that device-table pointer update is split
up into two 32bit writes in the IOMMU hardware. So updating it while
the IOMMU is enabled could have some nasty side effects.

The safe way to work around this is to always allocate the device-table
below 4G, including the old device-table in normal kernel and the
device-table used for copying the content of the old device-table in kdump
kernel. Meanwhile we need check if the address of old device-table is
above 4G because it might has been touched accidentally in corrupted
1st kernel.
Signed-off-by: NBaoquan He <bhe@redhat.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Firstly split the dev table entry copy into address translation part
and irq remapping part. Because these two parts could be enabled
independently.

Secondly do sanity check for address translation and irq remap of old
dev table entry separately.
Signed-off-by: NBaoquan He <bhe@redhat.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Here several things need be done:
- If iommu is pre-enabled in a normal kernel, just disable it and print
  warning.

- If any one of IOMMUs is not pre-enabled in kdump kernel, just continue
  as it does in normal kernel.

- If failed to copy dev table of old kernel, continue to proceed as
  it does in normal kernel.

- Only if all IOMMUs are pre-enabled and copy dev table is done well, free
  the dev table allocated in early_amd_iommu_init() and make amd_iommu_dev_table
  point to the copied one.

- Disable and Re-enable event/cmd buffer,  install the copied DTE table
  to reg, and detect and enable guest vapic.

- Flush all caches
Signed-off-by: NBaoquan He <bhe@redhat.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Add function copy_dev_tables to copy the old DEV table entries of the panicked
kernel to the new allocated device table. Since all iommus share the same device
table the copy only need be done one time. Here add a new global old_dev_tbl_cpy
to point to the newly allocated device table which the content of old device
table will be copied to. Besides, we also need to:

  - Check whether all IOMMUs actually use the same device table with the same size

  - Verify that the size of the old device table is the expected size.

  - Reserve the old domain id occupied in 1st kernel to avoid touching the old
    io-page tables. Then on-flight DMA can continue looking it up.

And also define MACRO DEV_DOMID_MASK to replace magic number 0xffffULL, it can be
reused in copy_dev_tables().
Signed-off-by: NBaoquan He <bhe@redhat.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

This reverts commit 54bd6357.

We still need the IO_PAGE_FAULT message to warn error after the
issue of on-flight dma in kdump kernel is fixed.
Signed-off-by: NBaoquan He <bhe@redhat.com>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>