Currently we saw a lot of "No irq handler" errors during hibernation, which
caused the system hang finally:

  ata4.00: qc timeout (cmd 0xec)
  ata4.00: failed to IDENTIFY (I/O error, err_mask=0x4)
  ata4.00: revalidation failed (errno=-5)
  ata4: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
  do_IRQ: 31.151 No irq handler for vector

According to above logs, there is an interrupt triggered and it is
dispatched to CPU31 with a vector number 151, but there is no handler for
it, thus this IRQ will not get acked and will cause an IRQ flood which
kills the system.  To be more specific, the 31.151 is an interrupt from the
AHCI host controller.

After some investigation, the reason why this issue is triggered is because
the thaw_noirq() function does not restore the MSI/MSI-X settings across
hibernation.

The scenario is illustrated below:

  1. Before hibernation, IRQ 34 is the handler for the AHCI device, which
     is bound to CPU31.

  2. Hibernation starts, the AHCI device is put into low power state.

  3. All the nonboot CPUs are put offline, so IRQ 34 has to be migrated to
     the last alive one - CPU0.

  4. After the snapshot has been created, all the nonboot CPUs are brought
     up again; IRQ 34 remains bound to CPU0.

  5. AHCI devices are put into D0.

  6. The snapshot is written to the disk.

The issue is triggered in step 6.  The AHCI interrupt should be delivered
to CPU0, however it is delivered to the original CPU31 instead, which
causes the "No irq handler" issue.

Ying Huang has provided a clue that, in step 3 it is possible that writing
to the register might not take effect as the PCI devices have been
suspended.

In step 3, the IRQ 34 affinity should be modified from CPU31 to CPU0, but
in fact it is not.  In __pci_write_msi_msg(), if the device is already in
low power state, the low level MSI message entry will not be updated but
cached.  During the device restore process after a normal suspend/resume,
pci_restore_msi_state() writes the cached MSI back to the hardware.

But this is not the case for hibernation.  pci_restore_msi_state() is not
currently called in pci_pm_thaw_noirq(), although pci_save_state() has
saved the necessary PCI cached information in pci_pm_freeze_noirq().

Restore the PCI status for the device during hibernation.  Otherwise the
status might be lost across hibernation (for example, settings for MSI,
MSI-X, ATS, ACS, IOV, etc.), which might cause problems during hibernation.
Suggested-by: NYing Huang <ying.huang@intel.com>
Suggested-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>
Signed-off-by: NChen Yu <yu.c.chen@intel.com>
[bhelgaas: changelog]
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Reviewed-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>
Cc: stable@vger.kernel.org
Cc: Len Brown <len.brown@intel.com>
Cc: Dan Williams <dan.j.williams@intel.com>
Cc: Rui Zhang <rui.zhang@intel.com>
Cc: Ying Huang <ying.huang@intel.com>

The PCI Power Management Spec, r1.2, sec 5.6.1, requires a 10 millisecond
delay when powering on a device, i.e., transitioning from state D3hot to
D0.

Apparently some devices require more time, and d1f9809e ("drm/radeon:
add quirk for d3 delay during switcheroo poweron for apple macbooks") added
an additional delay for the Radeon device in a MacBook Pro.  4807c5a8
("drm/radeon: add a PX quirk list") made the affected device more explicit.

Add a generic PCI quirk to increase the d3_delay.  This means we will use
the additional delay for *all* wakeups from D3, not just those initiated by
radeon_switcheroo_set_state().
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Reviewed-by: NAndreas Boll <andreas.boll.dev@gmail.com>
Acked-by: NAlex Deucher <alexander.deucher@amd.com>
CC: Maarten Lankhorst <maarten.lankhorst@canonical.com>

Some drivers - like i915 - may not support the system suspend direct
complete optimization due to differences in their runtime and system
suspend sequence.  Add a flag that when set resumes the device before
calling the driver's system suspend handlers which effectively disables
the optimization.

Needed by a future patch fixing suspend/resume on i915.

Suggested by Rafael.
Signed-off-by: NImre Deak <imre.deak@intel.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Acked-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>
Cc: stable@vger.kernel.org

Commit cc7b0d49 ("PCI: designware: Update PCI config space remap
function") made PCI configuration requests non-posted, which means we now
get a synchronous abort when the CFG space read to probe for downstream
devices times out.

Synchronous aborts need to be handled differently from the async aborts we
were getting before, in particular the PC needs to be advanced when
resolving the abort.  This is mostly a copy of what other PCI drivers do on
ARM to handle those aborts.

[bhelgaas: changelog, "Fixes"]
Fixes: cc7b0d49 ("PCI: designware: Update PCI config space remap function")
Tested-by: NFabio Estevam <fabio.estevam@nxp.com>
Tested-by: NPeter Senna Tschudin <peter.senna@collabora.com>
Signed-off-by: NLucas Stach <l.stach@pengutronix.de>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Acked-by: NRichard Zhu <hongxing.zhu@nxp.com>

When a switch endpoint is configured without NTB, the mmio_ntb registers
will read all zeros.  However, in corner case configurations where the
partition ID is not zero and NTB is not enabled, the code will have the
wrong partition ID and this causes the driver to use the wrong set of
drivers.  To fix this we simply take the partition ID from the system info
region.
Reported-by: NDingbao Chen <dingbao.chen@microsemi.com>
Signed-off-by: NLogan Gunthorpe <logang@deltatee.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

Convert from "cdev_add() + device_add()" to cdev_device_add(), and from
"device_del() + cdev_del()" to cdev_device_del().

[bhelgaas: changelog]
Signed-off-by: NLogan Gunthorpe <logang@deltatee.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

If NO_DMA=y:

    drivers/built-in.o: In function `__pci_epc_create':
    (.text+0xef4e): undefined reference to `bad_dma_ops'
    drivers/built-in.o: In function `pci_epc_add_epf':
    (.text+0xf676): undefined reference to `bad_dma_ops'
    drivers/built-in.o: In function `pci_epf_alloc_space':
    (.text+0xfa32): undefined reference to `bad_dma_ops'
    drivers/built-in.o: In function `pci_epf_free_space':
    (.text+0xfac4): undefined reference to `bad_dma_ops'

Add a dependency on HAS_DMA to fix this.
Signed-off-by: NGeert Uytterhoeven <geert@linux-m68k.org>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

Many PCI host controller drivers aren't prepared to have their devices
unbound from them forcefully (e.g., through /sys/.../<driver>/unbind), as
they don't provide any driver .remove callback, where they'd detach the
root bus, release resources, etc. Keeping the driver built in (i.e., not a
loadable module) is not enough; and providing no .remove callback just
means we don't do any teardown.

To rule out the possibility of unbinding a device via sysfs, we need to set
the ".suppress_bind_attrs" field.

I found the suspect drivers via the following search:

  git grep -l platform_driver $(git grep -L -e '\.remove' -e suppress_bind_attrs drivers/pci/)

Then I inspected them to ensure that
(a) they set up a PCI bus in their probe() and
(b) they don't have a remove() callback for undoing the setup
Suggested-by: NBjorn Helgaas <helgaas@kernel.org>
Signed-off-by: NBrian Norris <briannorris@chromium.org>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

According to errata i870, access to the PCIe slave port that are not 32-bit
aligned will result in incorrect mapping to TLP Address and Byte enable
fields.

Accessing non 32-bit aligned data causes incorrect data in the target
buffer if memcpy is used. Implement the workaround for this errata here.
Signed-off-by: NKishon Vijay Abraham I <kishon@ti.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

The PCIe controller integrated in dra7xx SoCs is capable of operating in
endpoint mode. Add endpoint mode support to dra7xx driver.
Signed-off-by: NKishon Vijay Abraham I <kishon@ti.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

No functional change. Split dra7xx_pcie_enable_interrupts() into
dra7xx_pcie_enable_wrapper_interrupts() and
dra7xx_pcie_enable_msi_interrupts() so that wrapper interrupts and MSI
interrupts can be enabled independently.  This is in preparation for adding
EP mode support to dra7xx driver since EP mode doesn't have to enable
msi_interrupts.
Signed-off-by: NKishon Vijay Abraham I <kishon@ti.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

Add endpoint mode support to designware driver. This uses the EP Core layer
introduced recently to add endpoint mode support.  *Any* function driver
can now use this designware device in order to achieve the EP
functionality.
Signed-off-by: NKishon Vijay Abraham I <kishon@ti.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

Trivial fix to spelling mistake in dev_err message
Signed-off-by: NColin Ian King <colin.king@canonical.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Acked-by: NRichard Zhu <hongxing.Zhu@nxp.com>

The ASUS M2N-LR should not trigger boot interrupt quirks although it
carries an Intel 6702PXH.  On this board the boot interrupt quirks cause
incorrect IRQ assignments and should be disabled.

Fixes: https://bugzilla.kernel.org/show_bug.cgi?id=43074Tested-by: NSolomon Peachy <pizza@shaftnet.org>
Signed-off-by: NStefan Assmann <sassmann@kpanic.de>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

PCI configuration space should be mapped with a memory region type that
generates on the CPU host bus non-posted write transations. Update the
driver to use the devm_ioremap_nopost* interface to make sure the correct
memory mappings for PCI configuration space are used.
Signed-off-by: NLorenzo Pieralisi <lorenzo.pieralisi@arm.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Cc: Rob Herring <robh@kernel.org>

PCI configuration space should be mapped with a memory region type that
generates on the CPU host bus non-posted write transations. Update the
driver to use the devm_pci_remap_cfg* interface to make sure the correct
memory mappings for PCI configuration space are used.
Signed-off-by: NLorenzo Pieralisi <lorenzo.pieralisi@arm.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Cc: Murali Karicheri <m-karicheri2@ti.com>

PCI configuration space should be mapped with a memory region type that
generates on the CPU host bus non-posted write transations. Update the
driver to use the devm_pci_remap_cfg* interface to make sure the correct
memory mappings for PCI configuration space are used.
Signed-off-by: NLorenzo Pieralisi <lorenzo.pieralisi@arm.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Cc: Mingkai Hu <mingkai.hu@freescale.com>
Cc: Minghuan Lian <minghuan.Lian@freescale.com>
Cc: Roy Zang <tie-fei.zang@freescale.com>

PCI configuration space should be mapped with a memory region type that
generates on the CPU host bus non-posted write transations. Update the
driver to use the devm_pci_remap_cfg* interface to make sure the correct
memory mappings for PCI configuration space are used.
Signed-off-by: NLorenzo Pieralisi <lorenzo.pieralisi@arm.com>
Signed-off-by: NGabriele Paoloni <gabriele.paoloni@huawei.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Cc: Zhou Wang <wangzhou1@hisilicon.com>

PCI configuration space should be mapped with a memory region type that
generates on the CPU host bus non-posted write transations. Update the
driver to use correct memory mapping attributes to map config space
regions to enforce configuration space non-posted writes behaviour.
Signed-off-by: NLorenzo Pieralisi <lorenzo.pieralisi@arm.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Cc: Thierry Reding <thierry.reding@gmail.com>

PCI configuration space should be mapped with a memory region type that
generates on the CPU host bus non-posted write transations. Update the
driver to use the devm_pci_remap_cfg* interface to make sure the correct
memory mappings for PCI configuration space are used.
Signed-off-by: NLorenzo Pieralisi <lorenzo.pieralisi@arm.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Cc: Tanmay Inamdar <tinamdar@apm.com>

PCI configuration space should be mapped with a memory region type that
generates on the CPU host bus non-posted write transations. Update the
driver to use the devm_pci_remap_cfg* interface to make sure the correct
memory mappings for PCI configuration space are used.
Signed-off-by: NLorenzo Pieralisi <lorenzo.pieralisi@arm.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>

PCI configuration space should be mapped with a memory region type that
generates on the CPU host bus non-posted write transations. Update the
driver to use the devm_pci_remap_cfg* interface to make sure the correct
memory mappings for PCI configuration space are used.
Signed-off-by: NLorenzo Pieralisi <lorenzo.pieralisi@arm.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Cc: Jingoo Han <jingoohan1@gmail.com>
Cc: Joao Pinto <Joao.Pinto@synopsys.com>

PCI configuration space should be mapped with a memory region type that
generates on the CPU host bus non-posted write transations. Update the
driver to use the devm_pci_remap_cfg* interface to make sure the correct
memory mappings for PCI configuration space are used.
Signed-off-by: NLorenzo Pieralisi <lorenzo.pieralisi@arm.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Cc: Ray Jui <rjui@broadcom.com>
Cc: Jon Mason <jonmason@broadcom.com>

PCI configuration space should be mapped with a memory region type that
generates on the CPU host bus non-posted write transations. Update the
driver to use the devm_pci_remap_cfg* interface to make sure the correct
memory mappings for PCI configuration space are used.
Signed-off-by: NLorenzo Pieralisi <lorenzo.pieralisi@arm.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Cc: Stanimir Varbanov <svarbanov@mm-sol.com>

PCI configuration space should be mapped with a memory region type that
generates on the CPU host bus non-posted write transations. Update the
driver to use the devm_pci_remap_cfg* interface to make sure the correct
memory mappings for PCI configuration space are used.
Signed-off-by: NLorenzo Pieralisi <lorenzo.pieralisi@arm.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Cc: Wenrui Li <wenrui.li@rock-chips.com>
Cc: Shawn Lin <shawn.lin@rock-chips.com>

PCI configuration space should be mapped with a memory region type that
generate on the CPU host bus non-posted write transations. Update the
driver to use the devm_pci_remap_cfg* interface to make sure the correct
memory mappings for PCI configuration space are used.
Signed-off-by: NLorenzo Pieralisi <lorenzo.pieralisi@arm.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Cc: Pratyush Anand <pratyush.anand@gmail.com>

PCI configuration space should be mapped with a memory region type that
generates on the CPU host bus non-posted write transations. Update the
driver to use the devm_pci_remap_cfg* interface to make sure the correct
memory mappings for PCI configuration space are used.
Signed-off-by: NLorenzo Pieralisi <lorenzo.pieralisi@arm.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Cc: Bharat Kumar Gogada <bharat.kumar.gogada@xilinx.com>
Cc: Michal Simek <michal.simek@xilinx.com>

PCI configuration space should be mapped with a memory region type that
generates on the CPU host bus non-posted write transations. Update the
driver to use the devm_pci_remap_cfg* interface to make sure the correct
memory mappings for PCI configuration space are used.
Signed-off-by: NLorenzo Pieralisi <lorenzo.pieralisi@arm.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Cc: Bharat Kumar Gogada <bharat.kumar.gogada@xilinx.com>
Cc: Michal Simek <michal.simek@xilinx.com>

The current ECAM kernel implementation uses ioremap() to map the ECAM
configuration space memory region; this is not safe in that on some
architectures the ioremap interface provides mappings that allow posted
write transactions. This, as highlighted in the PCIe specifications (4.0 -
Rev0.3, "Ordering Considerations for the Enhanced Configuration Address
Mechanism"), can create ordering issues for software because posted writes
transactions on the CPU host bus are non posted in the PCI express fabric.

Update the ioremap() interface to use pci_remap_cfgspace() whose mapping
attributes guarantee that non-posted writes transactions are issued for
memory writes within the ECAM memory mapped address region.
Signed-off-by: NLorenzo Pieralisi <lorenzo.pieralisi@arm.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Cc: Jayachandran C <jnair@caviumnetworks.com>

The introduction of the pci_remap_cfgspace() interface allows PCI host
controller drivers to map PCI config space through a dedicated kernel
interface. Current PCI host controller drivers use the devm_ioremap_*()
devres interfaces to map PCI configuration space regions so in order to
update them to the new pci_remap_cfgspace() mapping interface a new set of
devres interfaces should be implemented so that PCI host controller drivers
can make use of them.

Introduce two new functions in the PCI kernel layer and Devres
documentation:

- devm_pci_remap_cfgspace()
- devm_pci_remap_cfg_resource()

so that PCI host controller drivers can make use of them to map PCI
configuration space regions.
Signed-off-by: NLorenzo Pieralisi <lorenzo.pieralisi@arm.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Cc: Jonathan Corbet <corbet@lwn.net>

Now that we've exported pci_remap_iospace() and added proper remove()
support, there's no reason this can't be a loadable module.
Signed-off-by: NBrian Norris <briannorris@chromium.org>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Acked-by: NShawn Lin <shawn.lin@rock-chips.com>

These are useful for PCIe host drivers, and those drivers can be modules.

[bhelgaas: don't remove __weak; it's removed elsewhere]
Signed-off-by: NBrian Norris <briannorris@chromium.org>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Acked-by: NShawn Lin <shawn.lin@rock-chips.com>

Currently, if we try to unbind the platform device, the remove will
succeed, but the removal won't undo most of the registration, leaving
partially-configured PCI devices in the system.

This allows, for example, a simple 'lspci' to crash the system, as it will
try to touch the freed (via devm_*) driver structures, e.g., on RK3399:

  # echo f8000000.pcie > /sys/bus/platform/drivers/rockchip-pcie/unbind
  # lspci

So let's implement device remove().
Signed-off-by: NBrian Norris <briannorris@chromium.org>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Acked-by: NShawn Lin <shawn.lin@rock-chips.com>

Configuring DMA ops at probe time will allow deferring device probe when
the IOMMU isn't available yet. The dma_configure for the device is
now called from the generic device_attach callback just before the
bus/driver probe is called. This way, configuring the DMA ops for the
device would be called at the same place for all bus_types, hence the
deferred probing mechanism should work for all buses as well.

pci_bus_add_devices    (platform/amba)(_device_create/driver_register)
       |                         |
pci_bus_add_device     (device_add/driver_register)
       |                         |
device_attach           device_initial_probe
       |                         |
__device_attach_driver    __device_attach_driver
       |
driver_probe_device
       |
really_probe
       |
dma_configure

Similarly on the device/driver_unregister path __device_release_driver is
called which inturn calls dma_deconfigure.

This patch changes the dma ops configuration to probe time for
both OF and ACPI based platform/amba/pci bus devices.
Tested-by: NMarek Szyprowski <m.szyprowski@samsung.com>
Tested-by: NHanjun Guo <hanjun.guo@linaro.org>
Reviewed-by: NRobin Murphy <robin.murphy@arm.com>
Acked-by: NRob Herring <robh@kernel.org>
Acked-by: Bjorn Helgaas <bhelgaas@google.com> (drivers/pci part)
Acked-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>
Signed-off-by: NSricharan R <sricharan@codeaurora.org>
Signed-off-by: NJoerg Roedel <jroedel@suse.de>

Instead of copy & pasting and old version of the code.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

The 82599 quirk contained an outdated copy of the FLR code.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

Currently we opencode the FLR sequence in lots of place; export a core
helper instead.  We split out the probing for FLR support as all the
non-core callers already know their hardware.

Note that in the new pci_has_flr() function the quirk check has been moved
before the capability check as there is no point in reading the capability
in this case.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

Sometimes it is not desirable to bind SR-IOV VFs to drivers.  This can save
host side resource usage by VF instances that will be assigned to VMs.

Add a new PCI sysfs interface "sriov_drivers_autoprobe" to control that
from the PF.  To modify it, echo 0/n/N (disable probe) or 1/y/Y (enable
probe) to:

  /sys/bus/pci/devices/<DOMAIN:BUS:DEVICE.FUNCTION>/sriov_drivers_autoprobe

Note that this must be done before enabling VFs.  The change will not take
effect if VFs are already enabled.  Simply, one can disable VFs by setting
sriov_numvfs to 0, choose whether to probe or not, and then re-enable the
VFs by restoring sriov_numvfs.

[bhelgaas: changelog, ABI doc]
Signed-off-by: NBodong Wang <bodong@mellanox.com>
Signed-off-by: NEli Cohen <eli@mellanox.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Reviewed-by: NGavin Shan <gwshan@linux.vnet.ibm.com>
Reviewed-by: NAlex Williamson <alex.williamson@redhat.com>

This will need to call into an arch-provided pci_iobar_pfn() function.
Signed-off-by: NDavid Woodhouse <dwmw@amazon.co.uk>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

Starting to leave behind the legacy of the pci_mmap_page_range() interface
which takes "user-visible" BAR addresses.  This takes just the resource and
offset.

For now, both APIs coexist and depending on the platform, one is
implemented as a wrapper around the other.
Signed-off-by: NDavid Woodhouse <dwmw@amazon.co.uk>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>