pcie_downstream_port() is useful in other places where code needs to
determine whether the PCIe port is downstream so make it available outside
of access.c.

Link: https://lore.kernel.org/r/20190822085553.62697-1-mika.westerberg@linux.intel.comSigned-off-by: NMika Westerberg <mika.westerberg@linux.intel.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Reviewed-by: NAndy Shevchenko <andriy.shevchenko@linux.intel.com>

Move the ACPI-specific structs hpx_type0, hpx_type1, hpx_type2 and
hpx_type3 to drivers/pci/pci-acpi.c as they are not used anywhere else.
Then remove the struct hotplug_program_ops that has been shared between
drivers/pci/probe.c and drivers/pci/pci-acpi.c from drivers/pci/pci.h as it
is no longer needed.

The struct hotplug_program_ops was added by 87fcf12e ("PCI/ACPI: Remove
the need for 'struct hotplug_params'") and replaced previously used struct
hotplug_params enabling the support for the _HPX Type 3 Setting Record that
was added by f873c51a ("PCI/ACPI: Implement _HPX Type 3 Setting
Record").

The new struct allowed for the static functions such program_hpx_type0(),
program_hpx_type1(), etc., from the drivers/pci/probe.c to be called from
the function pci_acpi_program_hp_params() in the drivers/pci/pci-acpi.c.

Previously a programming of _HPX Type 0 was as follows:

  drivers/pci/probe.c:

    program_hpx_type0()
    ...
    pci_configure_device()
      hp_ops = {
        .program_type0 = program_hpx_type0,
        ...
      }
      pci_acpi_program_hp_params(&hp_ops)

  drivers/pci/pci-acpi.c:

    pci_acpi_program_hp_params(&hp_ops)
      acpi_run_hpx(hp_ops)
        decode_type0_hpx_record()
          hp_ops->program_type0     # program_hpx_type0() called via hp_ops

After the ACPI-specific functions, structs, enums, etc., have been moved to
drivers/pci/pci-acpi.c there is no need for the hotplug_program_ops as all
of the _HPX Type 0, 1, 2 and 3 are directly accessible.

Link: https://lore.kernel.org/r/20190827094951.10613-4-kw@linux.comSigned-off-by: NKrzysztof Wilczynski <kw@linux.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

Move program_hpx_type0(), program_hpx_type1(), etc., and enums
hpx_type3_dev_type, hpx_type3_fn_type and hpx_type3_cfg_loc to
drivers/pci/pci-acpi.c as these functions and enums are ACPI-specific.

Move structs hpx_type0, hpx_type1, hpx_type2 and hpx_type3 to
drivers/pci/pci.h as these are shared between drivers/pci/pci-acpi.c and
drivers/pci/probe.c.

Link: https://lore.kernel.org/r/20190827094951.10613-3-kw@linux.comSigned-off-by: NKrzysztof Wilczynski <kw@linux.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

These interfaces:

  void pci_set_of_node(struct pci_dev *dev);
  void pci_release_of_node(struct pci_dev *dev);
  void pci_set_bus_of_node(struct pci_bus *bus);
  void pci_release_bus_of_node(struct pci_bus *bus);

are only used in drivers/pci/ and do not need to be seen by the rest of the
kernel.  Move them to drivers/pci/pci.h so they're private to the PCI
subsystem.

Link: https://lore.kernel.org/r/20190724233848.73327-12-skunberg.kelsey@gmail.comSigned-off-by: NKelsey Skunberg <skunberg.kelsey@gmail.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

This interface:

  int pci_enable_ptm(struct pci_dev *dev, u8 *granularity);

is only used in drivers/pci/ and does not need to be seen by the rest of
the kernel.  Move it to drivers/pci/pci.h so it's private to the PCI
subsystem.

Link: https://lore.kernel.org/r/20190724233848.73327-11-skunberg.kelsey@gmail.comSigned-off-by: NKelsey Skunberg <skunberg.kelsey@gmail.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

These interfaces:

  void pcie_set_ecrc_checking(struct pci_dev *dev);
  void pcie_ecrc_get_policy(char *str);

are only used in drivers/pci/ and do not need to be seen by the rest of the
kernel.  Move them to drivers/pci/pci.h so they're private to the PCI
subsystem.

Link: https://lore.kernel.org/r/20190724233848.73327-10-skunberg.kelsey@gmail.comSigned-off-by: NKelsey Skunberg <skunberg.kelsey@gmail.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

This interface:

  void pci_ats_init(struct pci_dev *dev);

is only used in drivers/pci/ and does not need to be seen by the rest of
the kernel.  Move it to drivers/pci/pci.h so it's private to the PCI
subsystem.

Link: https://lore.kernel.org/r/20190724233848.73327-9-skunberg.kelsey@gmail.comSigned-off-by: NKelsey Skunberg <skunberg.kelsey@gmail.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

This interface:

  void pcie_update_link_speed(struct pci_bus *bus, u16 link_status);

is only used in drivers/pci/ and does not need to be seen by the rest of
the kernel.  Move it to drivers/pci/pci.h so it's private to the PCI
subsystem.

Link: https://lore.kernel.org/r/20190724233848.73327-8-skunberg.kelsey@gmail.comSigned-off-by: NKelsey Skunberg <skunberg.kelsey@gmail.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

These interfaces:

  struct pci_bus *pci_bus_get(struct pci_bus *bus);
  void pci_bus_put(struct pci_bus *bus);

are only used in drivers/pci/ and do not need to be seen by the rest of the
kernel.  Move them to drivers/pci/pci.h so they're private to the PCI
subsystem.

Link: https://lore.kernel.org/r/20190724233848.73327-7-skunberg.kelsey@gmail.comSigned-off-by: NKelsey Skunberg <skunberg.kelsey@gmail.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

These symbols:

  extern unsigned long pci_hotplug_io_size;
  extern unsigned long pci_hotplug_mem_size;
  extern unsigned long pci_hotplug_bus_size;

are only used in drivers/pci/ and do not need to be seen by the rest of the
kernel.  Move them to drivers/pci/pci.h so they're private to the PCI
subsystem.

Link: https://lore.kernel.org/r/20190724233848.73327-6-skunberg.kelsey@gmail.comSigned-off-by: NKelsey Skunberg <skunberg.kelsey@gmail.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

These Virtual Channel interfaces:

  int pci_save_vc_state(struct pci_dev *dev);
  void pci_restore_vc_state(struct pci_dev *dev);
  void pci_allocate_vc_save_buffers(struct pci_dev *dev);

are only used in drivers/pci/ and do not need to be seen by the rest of the
kernel.  Move them to drivers/pci/pci.h so they're private to the PCI
subsystem.

Link: https://lore.kernel.org/r/20190724233848.73327-5-skunberg.kelsey@gmail.comSigned-off-by: NKelsey Skunberg <skunberg.kelsey@gmail.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

These interfaces:

  struct device *pci_get_host_bridge_device(struct pci_dev *dev);
  void pci_put_host_bridge_device(struct device *dev);

are only used in drivers/pci/ and do not need to be seen by the rest of the
kernel.  Move them to drivers/pci/pci.h so they're private to the PCI
subsystem.

Link: https://lore.kernel.org/r/20190724233848.73327-4-skunberg.kelsey@gmail.comSigned-off-by: NKelsey Skunberg <skunberg.kelsey@gmail.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

These interfaces:

  bool pci_check_pme_status(struct pci_dev *dev);
  void pci_pme_wakeup_bus(struct pci_bus *bus);

are only used in drivers/pci/ and do not need to be seen by the rest of the
kernel.  Move them to drivers/pci/pci.h so they're private to the PCI
subsystem.

Link: https://lore.kernel.org/r/20190724233848.73327-3-skunberg.kelsey@gmail.comSigned-off-by: NKelsey Skunberg <skunberg.kelsey@gmail.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

These delay time definitions:

  #define PCI_PM_D2_DELAY         200
  #define PCI_PM_D3_WAIT          10
  #define PCI_PM_D3COLD_WAIT      100
  #define PCI_PM_BUS_WAIT         50

are only used in drivers/pci/ and do not need to be seen by the rest of the
kernel.  Move them to drivers/pci/pci.h so they're private to the PCI
subsystem.

Link: https://lore.kernel.org/r/20190724233848.73327-2-skunberg.kelsey@gmail.comSigned-off-by: NKelsey Skunberg <skunberg.kelsey@gmail.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

Revert 975bb8b4 ("PCI/IOV: Use VF0 cached config space size for other
VFs"), which attempted to cache the config space size from the first VF to
re-use for subsequent VFs.

The cached value was determined prior to discovering the PCIe capability on
the VF, which resulted in the first VF reporting the correct config space
size (4K), as it has a special case through pci_cfg_space_size(), while all
the other VFs only reported 256 bytes.  As this was only a performance
optimization, we're better off without it.

Fixes: 975bb8b4 ("PCI/IOV: Use VF0 cached config space size for other VFs")
Link: https://lore.kernel.org/r/156046663197.29869.3633634445109057665.stgit@gimli.homeSigned-off-by: NAlex Williamson <alex.williamson@redhat.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Cc: KarimAllah Ahmed <karahmed@amazon.de>
Cc: Kuppuswamy Sathyanarayanan <sathyanarayanan.kuppuswamy@linux.intel.com>
Cc: Hao Zheng <yinhe@linux.alibaba.com>

In pci_pm_complete() there are checks to decide whether or not to
resume devices that were left in runtime-suspend during the preceding
system-wide transition into a sleep state.  They involve checking the
current power state of the device and comparing it with the power
state of it set before the preceding system-wide transition, but the
platform component of the device's power state is not handled
correctly in there.

Namely, on platforms with ACPI, the device power state information
needs to be updated with care, so that the reference counters of
power resources used by the device (if any) are set to ensure that
the refreshed power state of it will be maintained going forward.

To that end, introduce a new ->refresh_state() platform PM callback
for PCI devices, for asking the platform to refresh the device power
state data and ensure that the corresponding power state will be
maintained going forward, make it invoke acpi_device_update_power()
(for devices with ACPI PM) on platforms with ACPI and make
pci_pm_complete() use it, through a new pci_refresh_power_state()
wrapper function.

Fixes: a0d2a959 (PCI: Avoid unnecessary resume after direct-complete)
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>
Reviewed-by: NMika Westerberg <mika.westerberg@linux.intel.com>

Currently Linux does not follow PCIe spec regarding the required delays
after reset. A concrete example is a Thunderbolt add-in-card that
consists of a PCIe switch and two PCIe endpoints:

  +-1b.0-[01-6b]----00.0-[02-6b]--+-00.0-[03]----00.0 TBT controller
                                  +-01.0-[04-36]-- DS hotplug port
                                  +-02.0-[37]----00.0 xHCI controller
                                  \-04.0-[38-6b]-- DS hotplug port

The root port (1b.0) and the PCIe switch downstream ports are all PCIe
gen3 so they support 8GT/s link speeds.

We wait for the PCIe hierarchy to enter D3cold (runtime):

  pcieport 0000:00:1b.0: power state changed by ACPI to D3cold

When it wakes up from D3cold, according to the PCIe 4.0 section 5.8 the
PCIe switch is put to reset and its power is re-applied. This means that
we must follow the rules in PCIe 4.0 section 6.6.1.

For the PCIe gen3 ports we are dealing with here, the following applies:

  With a Downstream Port that supports Link speeds greater than 5.0
  GT/s, software must wait a minimum of 100 ms after Link training
  completes before sending a Configuration Request to the device
  immediately below that Port. Software can determine when Link training
  completes by polling the Data Link Layer Link Active bit or by setting
  up an associated interrupt (see Section 6.7.3.3).

Translating this into the above topology we would need to do this (DLLLA
stands for Data Link Layer Link Active):

  pcieport 0000:00:1b.0: wait for 100ms after DLLLA is set before access to 0000:01:00.0
  pcieport 0000:02:00.0: wait for 100ms after DLLLA is set before access to 0000:03:00.0
  pcieport 0000:02:02.0: wait for 100ms after DLLLA is set before access to 0000:37:00.0

I've instrumented the kernel with additional logging so we can see the
actual delays the kernel performs:

  pcieport 0000:00:1b.0: power state changed by ACPI to D0
  pcieport 0000:00:1b.0: waiting for D3cold delay of 100 ms
  pcieport 0000:00:1b.0: waking up bus
  pcieport 0000:00:1b.0: waiting for D3hot delay of 10 ms
  pcieport 0000:00:1b.0: restoring config space at offset 0x2c (was 0x60, writing 0x60)
  ...
  pcieport 0000:00:1b.0: PME# disabled
  pcieport 0000:01:00.0: restoring config space at offset 0x3c (was 0x1ff, writing 0x201ff)
  ...
  pcieport 0000:01:00.0: PME# disabled
  pcieport 0000:02:00.0: restoring config space at offset 0x3c (was 0x1ff, writing 0x201ff)
  ...
  pcieport 0000:02:00.0: PME# disabled
  pcieport 0000:02:01.0: restoring config space at offset 0x3c (was 0x1ff, writing 0x201ff)
  ...
  pcieport 0000:02:01.0: restoring config space at offset 0x4 (was 0x100000, writing 0x100407)
  pcieport 0000:02:01.0: PME# disabled
  pcieport 0000:02:02.0: restoring config space at offset 0x3c (was 0x1ff, writing 0x201ff)
  ...
  pcieport 0000:02:02.0: PME# disabled
  pcieport 0000:02:04.0: restoring config space at offset 0x3c (was 0x1ff, writing 0x201ff)
  ...
  pcieport 0000:02:04.0: PME# disabled
  pcieport 0000:02:01.0: PME# enabled
  pcieport 0000:02:01.0: waiting for D3hot delay of 10 ms
  pcieport 0000:02:04.0: PME# enabled
  pcieport 0000:02:04.0: waiting for D3hot delay of 10 ms
  thunderbolt 0000:03:00.0: restoring config space at offset 0x14 (was 0x0, writing 0x8a040000)
  ...
  thunderbolt 0000:03:00.0: PME# disabled
  xhci_hcd 0000:37:00.0: restoring config space at offset 0x10 (was 0x0, writing 0x73f00000)
  ...
  xhci_hcd 0000:37:00.0: PME# disabled

For the switch upstream port (01:00.0) we wait for 100ms but not taking
into account the DLLLA requirement. We then wait 10ms for D3hot -> D0
transition of the root port and the two downstream hotplug ports. This
means that we deviate from what the spec requires.

Performing the same check for system sleep (s2idle) transitions we can
see following when resuming from s2idle:

  pcieport 0000:00:1b.0: power state changed by ACPI to D0
  pcieport 0000:00:1b.0: restoring config space at offset 0x2c (was 0x60, writing 0x60)
  ...
  pcieport 0000:01:00.0: restoring config space at offset 0x3c (was 0x1ff, writing 0x201ff)
  ...
  pcieport 0000:02:02.0: restoring config space at offset 0x3c (was 0x1ff, writing 0x201ff)
  pcieport 0000:02:02.0: restoring config space at offset 0x2c (was 0x0, writing 0x0)
  pcieport 0000:02:01.0: restoring config space at offset 0x3c (was 0x1ff, writing 0x201ff)
  pcieport 0000:02:04.0: restoring config space at offset 0x3c (was 0x1ff, writing 0x201ff)
  pcieport 0000:02:02.0: restoring config space at offset 0x28 (was 0x0, writing 0x0)
  pcieport 0000:02:00.0: restoring config space at offset 0x3c (was 0x1ff, writing 0x201ff)
  pcieport 0000:02:02.0: restoring config space at offset 0x24 (was 0x10001, writing 0x1fff1)
  pcieport 0000:02:01.0: restoring config space at offset 0x2c (was 0x0, writing 0x60)
  pcieport 0000:02:02.0: restoring config space at offset 0x20 (was 0x0, writing 0x73f073f0)
  pcieport 0000:02:04.0: restoring config space at offset 0x2c (was 0x0, writing 0x60)
  pcieport 0000:02:01.0: restoring config space at offset 0x28 (was 0x0, writing 0x60)
  pcieport 0000:02:00.0: restoring config space at offset 0x2c (was 0x0, writing 0x0)
  pcieport 0000:02:02.0: restoring config space at offset 0x1c (was 0x101, writing 0x1f1)
  pcieport 0000:02:04.0: restoring config space at offset 0x28 (was 0x0, writing 0x60)
  pcieport 0000:02:01.0: restoring config space at offset 0x24 (was 0x10001, writing 0x1ff10001)
  pcieport 0000:02:00.0: restoring config space at offset 0x28 (was 0x0, writing 0x0)
  pcieport 0000:02:02.0: restoring config space at offset 0x18 (was 0x0, writing 0x373702)
  pcieport 0000:02:04.0: restoring config space at offset 0x24 (was 0x10001, writing 0x49f12001)
  pcieport 0000:02:01.0: restoring config space at offset 0x20 (was 0x0, writing 0x73e05c00)
  pcieport 0000:02:00.0: restoring config space at offset 0x24 (was 0x10001, writing 0x1fff1)
  pcieport 0000:02:04.0: restoring config space at offset 0x20 (was 0x0, writing 0x89f07400)
  pcieport 0000:02:01.0: restoring config space at offset 0x1c (was 0x101, writing 0x5151)
  pcieport 0000:02:00.0: restoring config space at offset 0x20 (was 0x0, writing 0x8a008a00)
  pcieport 0000:02:02.0: restoring config space at offset 0xc (was 0x10000, writing 0x10020)
  pcieport 0000:02:04.0: restoring config space at offset 0x1c (was 0x101, writing 0x6161)
  pcieport 0000:02:01.0: restoring config space at offset 0x18 (was 0x0, writing 0x360402)
  pcieport 0000:02:00.0: restoring config space at offset 0x1c (was 0x101, writing 0x1f1)
  pcieport 0000:02:04.0: restoring config space at offset 0x18 (was 0x0, writing 0x6b3802)
  pcieport 0000:02:02.0: restoring config space at offset 0x4 (was 0x100000, writing 0x100407)
  pcieport 0000:02:00.0: restoring config space at offset 0x18 (was 0x0, writing 0x30302)
  pcieport 0000:02:01.0: restoring config space at offset 0xc (was 0x10000, writing 0x10020)
  pcieport 0000:02:04.0: restoring config space at offset 0xc (was 0x10000, writing 0x10020)
  pcieport 0000:02:00.0: restoring config space at offset 0xc (was 0x10000, writing 0x10020)
  pcieport 0000:02:01.0: restoring config space at offset 0x4 (was 0x100000, writing 0x100407)
  pcieport 0000:02:04.0: restoring config space at offset 0x4 (was 0x100000, writing 0x100407)
  pcieport 0000:02:00.0: restoring config space at offset 0x4 (was 0x100000, writing 0x100407)
  xhci_hcd 0000:37:00.0: restoring config space at offset 0x10 (was 0x0, writing 0x73f00000)
  ...
  thunderbolt 0000:03:00.0: restoring config space at offset 0x14 (was 0x0, writing 0x8a040000)

This is even worse. None of the mandatory delays are performed. If this
would be S3 instead of s2idle then according to PCI FW spec 3.2 section
4.6.8.  there is a specific _DSM that allows the OS to skip the delays
but this platform does not provide the _DSM and does not go to S3 anyway
so no firmware is involved that could already handle these delays.

In this particular Intel Coffee Lake platform these delays are not
actually needed because there is an additional delay as part of the ACPI
power resource that is used to turn on power to the hierarchy but since
that additional delay is not required by any of standards (PCIe, ACPI)
it is not present in the Intel Ice Lake, for example where missing the
mandatory delays causes pciehp to start tearing down the stack too early
(links are not yet trained).

For this reason, change the PCIe portdrv PM resume hooks so that they
perform the mandatory delays before the downstream component gets
resumed. We perform the delays before port services are resumed because
otherwise pciehp might find that the link is not up (even if it is just
training) and tears-down the hierarchy.
Signed-off-by: NMika Westerberg <mika.westerberg@linux.intel.com>
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

The code in pci_dev_keep_suspended() is relatively hard to follow due
to the negative checks in it and in its callers and the function has
a possible side-effect (disabling the PME) which doesn't really match
its role.

For this reason, move the PME disabling from pci_dev_keep_suspended()
to a separate function and change the semantics (and name) of the
rest of it, so that 'true' is returned when the device needs to be
resumed (and not the other way around).  Change the callers of
pci_dev_keep_suspended() accordingly.

While at it, make the code flow in pci_pm_poweroff() reflect the
pci_pm_suspend() more closely to avoid arbitrary differences between
them.

This is a cosmetic change with no intention to alter behavior.
Signed-off-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>
Reviewed-by: NMika Westerberg <mika.westerberg@linux.intel.com>

Commit 60ed982a ("PCI/AER: Move internal declarations to
drivers/pci/pci.h") changed pci_aer_init() to return "void", but didn't
change the stub for when CONFIG_PCIEAER isn't enabled.  Change the stub to
match.

Fixes: 60ed982a ("PCI/AER: Move internal declarations to drivers/pci/pci.h")
Signed-off-by: NJisheng Zhang <Jisheng.Zhang@synaptics.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
CC: stable@vger.kernel.org	# v4.19+

If a multi-function device's bandwidth is already limited when it is
enumerated, a message is logged only for function 0.  By contrast, when
downtraining occurs after enumeration, a message is logged for all
functions.  That's because the former uses pcie_report_downtraining(),
whereas the latter uses __pcie_print_link_status() (which doesn't filter
functions != 0).  I am seeing this happen on a MacBookPro9,1 with a GPU
(function 0) and an integrated HDA controller (function 1).

Avoid this incongruence by calling pcie_report_downtraining() in both
cases.
Signed-off-by: NLukas Wunner <lukas@wunner.de>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Reviewed-by: NAlexandru Gagniuc <alex.gagniuc@dellteam.com>

This file makes use of definitions provided in <linux/pci.h>.  This only
compiles when <linux/pci.h> is included beforehand, and creates a nasty
include dependency.  Instead, just include the correct file.
Signed-off-by: NAlexandru Gagniuc <mr.nuke.me@gmail.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

Cache the config space size from VF0 and use it for all other VFs instead
of reading it from the config space of each VF.  We assume that it will be
the same across all associated VFs.

This is an optimization when enabling SR-IOV on a device with many VFs.
Signed-off-by: NKarimAllah Ahmed <karahmed@amazon.de>
[bhelgaas: use CONFIG_PCI_IOV (not CONFIG_PCI_ATS)]
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

In order to have better power management for Thunderbolt PCIe chains,
Windows enables power management for native PCIe hotplug ports if there is
the following ACPI _DSD attached to the root port:

  Name (_DSD, Package () {
      ToUUID ("6211e2c0-58a3-4af3-90e1-927a4e0c55a4"),
      Package () {
          Package () {"HotPlugSupportInD3", 1}
      }
  })

This is also documented in:

  https://docs.microsoft.com/en-us/windows-hardware/drivers/pci/dsd-for-pcie-root-ports#identifying-pcie-root-ports-supporting-hot-plug-in-d3

Do the same in Linux by introducing new firmware PM callback
(->bridge_d3()) and then implement it for ACPI based systems so that the
above property is checked.

There is one catch, though. The initial pci_dev->bridge_d3 is set before
the root port has ACPI companion bound (the device is not added to the PCI
bus either) so we need to look up the ACPI companion manually in that case
in acpi_pci_bridge_d3().
Signed-off-by: NMika Westerberg <mika.westerberg@linux.intel.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Reviewed-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

Bring surprise removals and permanent failures together so we no longer
need separate flags.  The implementation enforces that error handling will
not be able to override a surprise removal's permanent channel failure.
Signed-off-by: NKeith Busch <keith.busch@intel.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Reviewed-by: NSinan Kaya <okaya@kernel.org>

We don't need to be paranoid about the topology changing while handling an
error.  If the device has changed in a hotplug capable slot, we can rely on
the presence detection handling to react to a changing topology.

Restore the fatal error handling behavior that existed before merging DPC
with AER with 7e9084b3 ("PCI/AER: Handle ERR_FATAL with removal and
re-enumeration of devices").
Signed-off-by: NKeith Busch <keith.busch@intel.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Reviewed-by: NSinan Kaya <okaya@kernel.org>

The secondary bus reset may have link side effects that a hotplug capable
port may incorrectly react to.  Use the slot specific reset for hotplug
ports, fixing the undesirable link down-up handling during error
recovering.
Signed-off-by: NKeith Busch <keith.busch@intel.com>
[bhelgaas: fold in
https://lore.kernel.org/linux-pci/20180926152326.14821-1-keith.busch@intel.com
for issue reported by Stephen Rothwell <sfr@canb.auug.org.au>]
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Reviewed-by: NSinan Kaya <okaya@kernel.org>

This patch provides DPC save and restore capabilities.  This is necessary
for the driver to observe DPC events in the event the configuration space
needs to be restored after a reset.
Signed-off-by: NKeith Busch <keith.busch@intel.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Reviewed-by: NSinan Kaya <okaya@kernel.org>

When both ends of a PCIe Link are capable of a higher bandwidth than is
currently in use, the Link is said to be "downtrained".  A downtrained Link
may indicate hardware or configuration problems in the system, but it's
hard to identify such Links from userspace.

Refactor pcie_print_link_status() so it continues to always print PCIe
bandwidth information, as several NIC drivers desire.

Add a new internal __pcie_print_link_status() to emit a message only when a
device's bandwidth is constrained by the fabric and call it from the PCI
core for all devices, which identifies all downtrained Links.  It also
emits messages for a few cases that are technically not downtrained, such
as a x4 device in an open-ended x1 slot.
Signed-off-by: NAlexandru Gagniuc <mr.nuke.me@gmail.com>
[bhelgaas: changelog, move __pcie_print_link_status() declaration to
drivers/pci/, rename pcie_check_upstream_link() to
pcie_report_downtraining()]
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

Intel Sunrise Point (SPT) PCH hardware has an implementation of the ACS
bits that does not comply with the PCIe standard.  To deal with this we
need device-specific quirks to disable ACS redirection.

Add a new pci_dev_specific_disable_acs_redir() quirk and a new
.disable_acs_redir() function pointer for use by non-compliant devices.  No
functional change intended.
Signed-off-by: NLogan Gunthorpe <logang@deltatee.com>
[bhelgaas: split to separate patch, move
pci_dev_specific_disable_acs_redir() declarations to drivers/pci/pci.h]
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Reviewed-by: NAlex Williamson <alex.williamson@redhat.com>

Move declarations for these functions:

  pci_dev_specific_acs_enabled()
  pci_dev_specific_enable_acs()

from include/linux/pci.h to drivers/pci/pci.h because nothing outside the
PCI core needs to use them.
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

When a PCI device is detected, pdev->is_added is set to 1 and proc and
sysfs entries are created.

When the device is removed, pdev->is_added is checked for one and then
device is detached with clearing of proc and sys entries and at end,
pdev->is_added is set to 0.

is_added and is_busmaster are bit fields in pci_dev structure sharing same
memory location.

A strange issue was observed with multiple removal and rescan of a PCIe
NVMe device using sysfs commands where is_added flag was observed as zero
instead of one while removing device and proc,sys entries are not cleared.
This causes issue in later device addition with warning message
"proc_dir_entry" already registered.

Debugging revealed a race condition between the PCI core setting the
is_added bit in pci_bus_add_device() and the NVMe driver reset work-queue
setting the is_busmaster bit in pci_set_master().  As these fields are not
handled atomically, that clears the is_added bit.

Move the is_added bit to a separate private flag variable and use atomic
functions to set and retrieve the device addition state.  This avoids the
race because is_added no longer shares a memory location with is_busmaster.

Link: https://bugzilla.kernel.org/show_bug.cgi?id=200283Signed-off-by: NHari Vyas <hari.vyas@broadcom.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Reviewed-by: NLukas Wunner <lukas@wunner.de>
Acked-by: NMichael Ellerman <mpe@ellerman.id.au>

Clear the device status bits while handling both ERR_FATAL and ERR_NONFATAL
cases.
Signed-off-by: NOza Pawandeep <poza@codeaurora.org>
[bhelgaas: rename to pci_aer_clear_device_status(), declare internal to PCI
core instead of exposing it everywhere]
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

During recovery from fatal errors, we previously called
pci_cleanup_aer_uncorrect_error_status(), which cleared *all* uncorrectable
error status bits (both ERR_FATAL and ERR_NONFATAL).

Instead, call a new pci_aer_clear_fatal_status() that clears only the
ERR_FATAL bits (as indicated by the PCI_ERR_UNCOR_SEVER register).
Based-on-patch-by: NOza Pawandeep <poza@codeaurora.org>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

Rename pci_reset_bridge_secondary_bus() to pci_bridge_secondary_bus_reset()
and move the declaration from linux/pci.h to drivers/pci.h to be used
internally in PCI directory only.
Signed-off-by: NSinan Kaya <okaya@codeaurora.org>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

Add sysfs attributes to provide total and breakdown of the AERs seen,
into different type of correctable, fatal and nonfatal errors:

  /sys/bus/pci/devices/<dev>/aer_dev_correctable
  /sys/bus/pci/devices/<dev>/aer_dev_fatal
  /sys/bus/pci/devices/<dev>/aer_dev_nonfatal
Signed-off-by: NRajat Jain <rajatja@google.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

Define a structure to hold the AER statistics.  There are 2 groups of
statistics: dev_* counters that are to be collected for all AER capable
devices and rootport_* counters that are collected for all (AER capable)
rootports only.  Allocate and free this structure when device is added or
released (thus counters survive the lifetime of the device).
Signed-off-by: NRajat Jain <rajatja@google.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

Since pci_aer_init() and pci_no_aer() are used only internally, move their
declarations to the PCI internal header file.  Also, no one cares about
return value of pci_aer_init(), so make it void.
Signed-off-by: NRajat Jain <rajatja@google.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

Export some common AER functions and structures for other PCI core drivers
to use.  Since this is making the function externally visible inside the
PCI core, prepend "aer_" to the function name.
Signed-off-by: NKeith Busch <keith.busch@intel.com>
[bhelgaas: move AER declarations from linux/aer.h to drivers/pci/pci.h]
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Reviewed-by: NSinan Kaya <okaya@kernel.org>
Reviewed-by: NOza Pawandeep <poza@codeaurora.org>

Some IDT switches incorrectly flag an ACS Source Validation error on
completions for config read requests even though PCIe r4.0, sec 6.12.1.1,
says that completions are never affected by ACS Source Validation.  Here's
the text of IDT 89H32H8G3-YC, erratum #36:

  Item #36 - Downstream port applies ACS Source Validation to Completions
  Section 6.12.1.1 of the PCI Express Base Specification 3.1 states that
  completions are never affected by ACS Source Validation.  However,
  completions received by a downstream port of the PCIe switch from a
  device that has not yet captured a PCIe bus number are incorrectly
  dropped by ACS Source Validation by the switch downstream port.

  Workaround: Issue a CfgWr1 to the downstream device before issuing the
  first CfgRd1 to the device.  This allows the downstream device to capture
  its bus number; ACS Source Validation no longer stops completions from
  being forwarded by the downstream port.  It has been observed that
  Microsoft Windows implements this workaround already; however, some
  versions of Linux and other operating systems may not.

When doing the first config read to probe for a device, if the device is
behind an IDT switch with this erratum:

  1. Disable ACS Source Validation if enabled
  2. Wait for device to become ready to accept config accesses (by using
     the Config Request Retry Status mechanism)
  3. Do a config write to the endpoint
  4. Enable ACS Source Validation (if it was enabled to begin with)

The workaround suggested by IDT is basically only step 3, but we don't know
when the device is ready to accept config requests.  That means we need to
do config reads until we receive a non-Config Request Retry Status, which
means we need to disable ACS SV temporarily.
Signed-off-by: NJames Puthukattukaran <james.puthukattukaran@oracle.com>
[bhelgaas: changelog, clean up whitespace, fold in unused variable fix
from Anders Roxell <anders.roxell@linaro.org>]
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Reviewed-by: NAlex Williamson <alex.williamson@redhat.com>

Move early dump functionality into common code so that it is available for
all architectures.  No need to carry arch-specific reads around as the read
hooks are already initialized by the time pci_setup_device() is getting
called during scan.
Tested-by: NAndy Shevchenko <andy.shevchenko@gmail.com>
Signed-off-by: NSinan Kaya <okaya@codeaurora.org>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Reviewed-by: NAndy Shevchenko <andy.shevchenko@gmail.com>