Client VMD platforms have a software-triggered MSI-X vector 0 that will
not forward hardware-remapped MSI from the sub-device domain. This
causes an issue with VMD platforms that use AHCI behind VMD and have a
single MSI-X vector remapped to VMD vector 0. Add a VMD MSI-X vector
offset for these platforms.

Link: https://lore.kernel.org/r/20201102222223.92978-1-jonathan.derrick@intel.comTested-by: NJian-Hong Pan <jhp@endlessos.org>
Signed-off-by: NJon Derrick <jonathan.derrick@intel.com>
Signed-off-by: NLorenzo Pieralisi <lorenzo.pieralisi@arm.com>

The pci_save_state() call in vmd_suspend() can be performed by
pci_pm_suspend_irq(). This also allows VMD to benefit from the call into
pci_prepare_to_sleep().

The pci_restore_state() call in vmd_resume() was restoring state after
pci_pm_resume()::pci_restore_standard_config() had already restored state.
It's also been suspected that the config state should have been restored
before re-requesting IRQs instead of afterwards.

Remove the pci_save_state()/pci_restore_state() calls in
vmd_suspend()/vmd_resume() to allow proper flow through generic PCI core
Power Management code.

Link: https://lore.kernel.org/r/20200806210017.5654-1-jonathan.derrick@intel.comSigned-off-by: NJon Derrick <jonathan.derrick@intel.com>
Signed-off-by: NLorenzo Pieralisi <lorenzo.pieralisi@arm.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Cc: Kai-Heng Feng <kai.heng.feng@canonical.com>
Cc: You-Sheng Yang <vicamo.yang@canonical.com>

Move the IRQ allocation and SRCU initialization code to a new helper.  No
functional changes.

Link: https://lore.kernel.org/r/20200728194945.14126-5-jonathan.derrick@intel.comSigned-off-by: NJon Derrick <jonathan.derrick@intel.com>
Signed-off-by: NLorenzo Pieralisi <lorenzo.pieralisi@arm.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Reviewed-by: NAndy Shevchenko <andriy.shevchenko@intel.com>

Move the IRQ and MSI Domain configuration code to new helpers. No
functional changes.

Link: https://lore.kernel.org/r/20200728194945.14126-4-jonathan.derrick@intel.comSigned-off-by: NJon Derrick <jonathan.derrick@intel.com>
Signed-off-by: NLorenzo Pieralisi <lorenzo.pieralisi@arm.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Reviewed-by: NAndy Shevchenko <andriy.shevchenko@intel.com>

Move the bus offset configuration discovery code to a new helper.  Modify
the bus offset 2-bit decode switch to have a 0 case and a default error
case, just in case the field is expanded in future hardware.

Link: https://lore.kernel.org/r/20200728194945.14126-3-jonathan.derrick@intel.comSigned-off-by: NJon Derrick <jonathan.derrick@intel.com>
Signed-off-by: NLorenzo Pieralisi <lorenzo.pieralisi@arm.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Reviewed-by: NAndy Shevchenko <andriy.shevchenko@intel.com>

Move the guest-passthrough physical offset discovery code to a new helper.
No functional changes.

Link: https://lore.kernel.org/r/20200728194945.14126-2-jonathan.derrick@intel.comSigned-off-by: NJon Derrick <jonathan.derrick@intel.com>
Signed-off-by: NLorenzo Pieralisi <lorenzo.pieralisi@arm.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Reviewed-by: NAndy Shevchenko <andriy.shevchenko@intel.com>

Devices on the VMD bus use their own MSI irq domain, but it is not
distinguishable from regular PCI/MSI irq domains. This is required
to exclude VMD devices from getting the irq domain pointer set by
interrupt remapping.

Override the default bus token.
Signed-off-by: NThomas Gleixner <tglx@linutronix.de>
Acked-by: NBjorn Helgaas <bhelgaas@google.com>
Acked-by: NJoerg Roedel <jroedel@suse.de>
Link: https://lore.kernel.org/r/20200826112333.047315047@linutronix.de

VMD has it's own PCI/MSI interrupt domain which is not in any way depending
on the x86 vector domain. PCI devices behind VMD share the VMD MSIX vector
entries via a VMD specific message translation to the actual VMD MSIX
vector. The VMD device interrupt handler for the VMD MSIX vectors invokes
all interrupt handlers of the devices which share a vector.

Making the x86 vector domain the actual parent of the VMD irq domain is
pointless and actually counterproductive. When a device interrupt is
requested then it will activate the interrupt which traverses down the
hierarchy and consumes an interrupt vector in the vector domain which is
never used.

The domain is self contained and has no parent dependencies, so just hand
in NULL for the parent and be done with it.
Signed-off-by: NThomas Gleixner <tglx@linutronix.de>
Link: https://lore.kernel.org/r/20200826112330.928952181@linutronix.de

Commit 711419e5 ("irqdomain: Add the missing assignment of
domain->fwnode for named fwnode") unintentionally caused a dangling pointer
page fault issue on firmware nodes that were freed after IRQ domain
allocation. Commit e3beca48 fixed that dangling pointer issue by only
freeing the firmware node after an IRQ domain allocation failure. That fix
no longer frees the firmware node immediately, but leaves the firmware node
allocated after the domain is removed.

The firmware node must be kept around through irq_domain_remove, but should be
freed it afterwards.

Add the missing free operations after domain removal where where appropriate.

Fixes: e3beca48 ("irqdomain/treewide: Keep firmware node unconditionally allocated")
Signed-off-by: NJon Derrick <jonathan.derrick@intel.com>
Signed-off-by: NThomas Gleixner <tglx@linutronix.de>
Reviewed-by: NAndy Shevchenko <andriy.shevchenko@linux.intel.com>
Acked-by: Bjorn Helgaas <bhelgaas@google.com>	# drivers/pci
Cc: stable@vger.kernel.org
Link: https://lkml.kernel.org/r/1595363169-7157-1-git-send-email-jonathan.derrick@intel.com

Quite some non OF/ACPI users of irqdomains allocate firmware nodes of type
IRQCHIP_FWNODE_NAMED or IRQCHIP_FWNODE_NAMED_ID and free them right after
creating the irqdomain. The only purpose of these FW nodes is to convey
name information. When this was introduced the core code did not store the
pointer to the node in the irqdomain. A recent change stored the firmware
node pointer in irqdomain for other reasons and missed to notice that the
usage sites which do the alloc_fwnode/create_domain/free_fwnode sequence
are broken by this. Storing a dangling pointer is dangerous itself, but in
case that the domain is destroyed later on this leads to a double free.

Remove the freeing of the firmware node after creating the irqdomain from
all affected call sites to cure this.

Fixes: 711419e5 ("irqdomain: Add the missing assignment of domain->fwnode for named fwnode")
Reported-by: NAndy Shevchenko <andriy.shevchenko@linux.intel.com>
Signed-off-by: NThomas Gleixner <tglx@linutronix.de>
Acked-by: NBjorn Helgaas <bhelgaas@google.com>
Acked-by: NMarc Zyngier <maz@kernel.org>
Cc: stable@vger.kernel.org
Link: https://lkml.kernel.org/r/873661qakd.fsf@nanos.tec.linutronix.de

VMD device 28C0 natively assists guest passthrough of the VMD endpoint
through the use of shadow registers that provide Host Physical Addresses
to correctly assign bridge windows. These shadow registers are only
available if VMD config space register 0x70, bit 1 is set.

In order to support this mode in existing VMD devices which don't
natively support the shadow register, it was decided that the hypervisor
could offer the shadow registers in a vendor-specific PCI capability.

QEMU has been modified to create this vendor-specific capability and
supply the shadow membar registers for VMDs which don't natively support
this feature. This patch adds this mode and updates the supported device
list to allow this feature to be used on these VMDs.

Link: https://lore.kernel.org/r/20200528030240.16024-4-jonathan.derrick@intel.comSigned-off-by: NJon Derrick <jonathan.derrick@intel.com>
Signed-off-by: NLorenzo Pieralisi <lorenzo.pieralisi@arm.com>

Versions of VMD with the Host Physical Address shadow register use this
register to calculate the bus address offset needed to do guest
passthrough of the domain. This register shadows the Host Physical
Address registers including the resource type bits. After calculating
the offset, the extra resource type bits lead to the VMD resources being
over-provisioned at the front and under-provisioned at the back.

Example:
pci 10000:80:02.0: reg 0x10: [mem 0xf801fffc-0xf803fffb 64bit]

Expected:
pci 10000:80:02.0: reg 0x10: [mem 0xf8020000-0xf803ffff 64bit]

If other devices are mapped in the over-provisioned front, it could lead
to resource conflict issues with VMD or those devices.

Link: https://lore.kernel.org/r/20200528030240.16024-3-jonathan.derrick@intel.com
Fixes: a1a30170 ("PCI: vmd: Fix shadow offsets to reflect spec changes")
Signed-off-by: NJon Derrick <jonathan.derrick@intel.com>
Signed-off-by: NLorenzo Pieralisi <lorenzo.pieralisi@arm.com>

Add new VMD device IDs that require the bus restriction mode.
Signed-off-by: NSushma Kalakota <sushmax.kalakota@intel.com>
Signed-off-by: NJon Derrick <jonathan.derrick@intel.com>
Signed-off-by: NLorenzo Pieralisi <lorenzo.pieralisi@arm.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

Add new VMD device IDs that require the bus restriction mode.
Signed-off-by: NSushma Kalakota <sushmax.kalakota@intel.com>
Signed-off-by: NJon Derrick <jonathan.derrick@intel.com>
Signed-off-by: NLorenzo Pieralisi <lorenzo.pieralisi@arm.com>

Devices on the VMD domain use the VMD endpoint's requester ID and have been
relying on the VMD endpoint's DMA operations.  The problem with this was
that VMD domain devices would use the VMD endpoint's attributes when doing
DMA and IOMMU mapping.  We can be smarter about this by only using the VMD
endpoint when mapping and providing the correct child device's attributes
during DMA operations.

Remove the dma_map_ops redirect.

Link: https://lore.kernel.org/r/1579613871-301529-7-git-send-email-jonathan.derrick@intel.comSigned-off-by: NJon Derrick <jonathan.derrick@intel.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Reviewed-by: NKeith Busch <kbusch@kernel.org>
Acked-by: NLorenzo Pieralisi <lorenzo.pieralisi@arm.com>

Expose VMD's pci_dev pointer in struct pci_sysdata.  This will be used
indirectly by intel-iommu.c to find the correct domain.

Link: https://lore.kernel.org/r/1579613871-301529-3-git-send-email-jonathan.derrick@intel.com
[bhelgaas: commit log]
Signed-off-by: NJon Derrick <jonathan.derrick@intel.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Cc: Christoph Hellwig <hch@lst.de>

Adjust indentation from spaces to tab (+optional two spaces) as in
coding style with command like:

  $ sed -e 's/^        /\t/' -i */Kconfig

[bhelgaas: do same in vmd.c]
Link: https://lore.kernel.org/r/20191120134036.14502-1-krzk@kernel.orgSigned-off-by: NKrzysztof Kozlowski <krzk@kernel.org>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

This patch adds support for this VMD device which supports the bus
restriction mode.
Signed-off-by: NJon Derrick <jonathan.derrick@intel.com>
Signed-off-by: NLorenzo Pieralisi <lorenzo.pieralisi@arm.com>

VMD bus restrictions are required when IO fabric is multiplexed such
that VMD cannot use the entire bus range. This patch adds another bus
restriction decode bit that can be set by firmware to restrict the VMD
bus range to between 224-255.
Signed-off-by: NJon Derrick <jonathan.derrick@intel.com>
Signed-off-by: NLorenzo Pieralisi <lorenzo.pieralisi@arm.com>

The shadow offset scratchpad was moved to 0x2000-0x2010. Update the
location to get the correct shadow offset.

Fixes: 6788958e ("PCI: vmd: Assign membar addresses from shadow registers")
Signed-off-by: NJon Derrick <jonathan.derrick@intel.com>
Signed-off-by: NLorenzo Pieralisi <lorenzo.pieralisi@arm.com>
Cc: stable@vger.kernel.org # v5.2+

VMD maps child device config spaces to the VMD Config BAR linearly
regardless of the starting bus offset. Because of this, the config
address decode must ignore starting bus offsets when mapping the BDF to
the config space address.

Fixes: 2a5a9c9a ("PCI: vmd: Add offset to bus numbers if necessary")
Signed-off-by: NJon Derrick <jonathan.derrick@intel.com>
Signed-off-by: NLorenzo Pieralisi <lorenzo.pieralisi@arm.com>
Cc: stable@vger.kernel.org # v5.2+

Fix typos in drivers/pci.  Comment and whitespace changes only.
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Reviewed-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>
Reviewed-by: NRandy Dunlap <rdunlap@infradead.org>

We have supported per-device dma_map_ops in generic code for a long
time, and this symbol just guards the inclusion of the dma_map_ops
registry used for vmd.  Stop enabling it for anything but vmd.

No change in functionality intended.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Acked-by: NBjorn Helgaas <bhelgaas@google.com>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Link: http://lkml.kernel.org/r/20190410080220.21705-3-hch@lst.deSigned-off-by: NIngo Molnar <mingo@kernel.org>

In order to provide the most performance and/or compatible settings,
ensure VMD root buses observe the pcie bus tuning settings by
configuring those settings prior to adding the devices to the pcie tree.

This patch open-codes pci_rescan_bus() and configures the buses prior to
adding devices and attaching drivers.
Signed-off-by: NJon Derrick <jonathan.derrick@intel.com>
Signed-off-by: NLorenzo Pieralisi <lorenzo.pieralisi@arm.com>
Reviewed-by: NKeith Busch <keith.busch@intel.com>

With the bypass support for the direct mapping we might not always have
methods to call, so use the proper APIs instead.  The only downside is
that we will create two dma-debug entries for each mapping if
CONFIG_DMA_DEBUG is enabled.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Acked-by: NJesper Dangaard Brouer <brouer@redhat.com>
Tested-by: NJesper Dangaard Brouer <brouer@redhat.com>
Tested-by: NTony Luck <tony.luck@intel.com>

No users left except for vmd which just forwards it.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Acked-by: NLinus Torvalds <torvalds@linux-foundation.org>

The VMD removal path calls pci_stop_root_busi(), which tears down the pcie
tree, including detaching all of the attached drivers. During driver
detachment, devices may use pci_release_region() to release resources.
This path relies on the resource being accessible in resource tree.

By detaching the child domain from the parent resource domain prior to
stopping the bus, we are preventing the list traversal from finding the
resource to be freed. If we instead detach the resource after stopping
the bus, we will have properly freed the resource and detaching is
simply accounting at that point.

Without this order, the resource is never freed and is orphaned on VMD
removal, leading to a warning:

[  181.940162] Trying to free nonexistent resource <e5a10000-e5a13fff>

Fixes: 2c2c5c5c ("x86/PCI: VMD: Attach VMD resources to parent domain's resource tree")
Signed-off-by: NJon Derrick <jonathan.derrick@intel.com>
[lorenzo.pieralisi@arm.com: updated commit log]
Signed-off-by: NLorenzo Pieralisi <lorenzo.pieralisi@arm.com>
Reviewed-by: NKeith Busch <keith.busch@intel.com>

This save some duplication for ia64, and makes the interface more
general.  In the long run we want each dma_map_ops instance to fill this
out, but this will take a little more prep work.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Acked-by: NBenjamin Herrenschmidt <benh@kernel.crashing.org>

Devices with slow interrupt handlers are significantly harming
performance when their interrupt vector is shared with a fast device.

Create a class code white list for devices with known fast interrupt
handlers and let all other devices share a single vector so that they
don't interfere with performance.

At the moment, only the NVM Express class code is on the list, but more
may be added if VMD users desire to use other low-latency devices in
these domains.
Signed-off-by: NKeith Busch <keith.busch@intel.com>
[lorenzo.pieralisi@arm.com: changelog]
Signed-off-by: NLorenzo Pieralisi <lorenzo.pieralisi@arm.com>
Acked-by: NJon Derrick: <jonathan.derrick@intel.com>

Native PCI drivers for root complex devices were originally all in
drivers/pci/host/.  Some of these devices can also be operated in endpoint
mode.  Drivers for endpoint mode didn't seem to fit in the "host"
directory, so we put both the root complex and endpoint drivers in
per-device directories, e.g., drivers/pci/dwc/, drivers/pci/cadence/, etc.

These per-device directories contain trivial Kconfig and Makefiles and
clutter drivers/pci/.  Make a new drivers/pci/controllers/ directory and
collect all the device-specific drivers there.

No functional change intended.

Link: https://lkml.kernel.org/r/1520304202-232891-1-git-send-email-shawn.lin@rock-chips.comSigned-off-by: NShawn Lin <shawn.lin@rock-chips.com>
[bhelgaas: changelog]
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

Allow VMD devices with PCI id 8086:28c0 to bind to VMD driver.
Signed-off-by: NJon Derrick <jonathan.derrick@intel.com>
[lorenzo.pieralisi@arm.com: updated commit subject]
Signed-off-by: NLorenzo Pieralisi <lorenzo.pieralisi@arm.com>

Depending on platform configuration, certain VMD devices may have an
additional configuration option which specifies the range of bus numbers
allowed in a VMD PCIe domain. We determine this requirement by checking
the value of two vendor specific config registers in the VMD endpoint:

 VMCAP[0] | VMCONFIG[9:8] | Bus Numbers
----------------------------------------
    0     |       *       |     0-255
    1     |      00       |     0-127
    1     |      01       |   128-255
    1     |      10       |     0-255

This feature is also added as a bit in driver_data, to allow future
conforming device ids which support these features to be enabled through
sysfs new_id.
Signed-off-by: NJon Derrick <jonathan.derrick@intel.com>
[lorenzo.pieralisi@arm.com: updated commit subject]
Signed-off-by: NLorenzo Pieralisi <lorenzo.pieralisi@arm.com>

Certain VMD devices have registers within membar 2 which may shadow the
membar 1 and membar 2 addresses. These are intended to be used in
virtualization, where assigning a guest address wouldn't be translated
in the assignment to root port and child devices because the addresses
exist within the assignment message.

These values will only reflect the membars when enabled in the BIOS, as
determined by a register in the VMD device.

This patch declares this option as a bit in the pci id driver_data, so
that future conforming device ids can be enabled through sysfs new_id if
necessary.
Signed-off-by: NJon Derrick <jonathan.derrick@intel.com>
[lorenzo.pieralisi@arm.com: updated commit subject]
Signed-off-by: NLorenzo Pieralisi <lorenzo.pieralisi@arm.com>

Add the Intel VMD device ids to the pci id database and update the VMD
driver.
Signed-off-by: NJon Derrick <jonathan.derrick@intel.com>
Signed-off-by: NLorenzo Pieralisi <lorenzo.pieralisi@arm.com>

Add SPDX GPL-2.0 to all PCI files that specified the GPL version 2 license.

Remove the boilerplate GPL version 2 language, relying on the assertion in
b2441318 ("License cleanup: add SPDX GPL-2.0 license identifier to
files with no license") that the SPDX identifier may be used instead of the
full boilerplate text.
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Reviewed-by: NGreg Kroah-Hartman <gregkh@linuxfoundation.org>

VMD hardware has to share its vectors among child devices in its PCI
domain so we should allocate as many as possible rather than just ones
that can be affinitized.

pci_alloc_irq_vectors_affinity() limits the number of affinitized IRQs to
the number of present CPUs (see irq_calc_affinity_vectors()).  But we'd
prefer to have more vectors, even if they aren't distributed across the
CPUs, so use pci_alloc_irq_vectors() instead.
Reported-by: NBrad Goodman <Bradley.Goodman@dell.com>
Signed-off-by: NKeith Busch <keith.busch@intel.com>
[bhelgaas: add irq_calc_affinity_vectors() reference to changelog]
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

Free up the IRQs we request on the suspend path and reallocate them on the
resume path.

Fixes this error:

  CPU 111 disable failed: CPU has 9 vectors assigned and there are only 0 available.
  Error taking CPU111 down: -34
  Non-boot CPUs are not disabled
  Enabling non-boot CPUs ...
Signed-off-by: NScott Bauer <scott.bauer@intel.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Acked-by: NKeith Busch <keith.busch@intel.com>

We don't want slower IRQ handlers impacting faster devices that happen to
be assigned the same VMD interrupt vector. The driver was trying to
separate such devices by checking if MSI-X wasn't used, but really we just
don't want endpoint devices to share with bridges. Most bridges may use MSI
currently, so that criteria happened to work, but newer ones may use MSI-X,
so this patch explicitly checks the device type when choosing a vector.
Signed-off-by: NKeith Busch <keith.busch@intel.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

The driver has a special purpose for the VMD device's first IRQ, so this
one shouldn't be considered for IRQ affinity.
Signed-off-by: NKeith Busch <keith.busch@intel.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

Recent __call_srcu() changes have exposed that we need to cleanup SRCU
structures after pci_stop_root_bus() calls into vmd_msi_free().

Fixes: 3906b918 ("PCI: vmd: Use SRCU as a local RCU to prevent delaying global RCU")
Signed-off-by: NJon Derrick <jonathan.derrick@intel.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Acked-by: NKeith Busch <keith.busch@intel.com>
Cc: <stable@vger.kernel.org> # 4.11