Make pci_enable_ptm() accessible from the drivers.

Exposing this to the driver enables the driver to use the
'ptm_enabled' field of 'pci_dev' to check if PTM is enabled or not.

This reverts commit ac6c26da ("PCI: Make pci_enable_ptm() private").
Signed-off-by: NVinicius Costa Gomes <vinicius.gomes@intel.com>
Acked-by: NBjorn Helgaas <bhelgaas@google.com>
Signed-off-by: NTony Nguyen <anthony.l.nguyen@intel.com>

Fix kernel-doc formatting throughout drivers/pci and related include files.
No change to functionality intended.

Check for warnings:

  $ find include drivers/pci -type f -path "*pci*.[ch]" | xargs scripts/kernel-doc -none

[bhelgaas: squashed to one commit]
Link: https://lore.kernel.org/r/20210509030237.368540-1-kw@linux.com
Link: https://lore.kernel.org/r/20210703151306.1922450-1-kw@linux.com
Link: https://lore.kernel.org/r/20210703151306.1922450-2-kw@linux.com
Link: https://lore.kernel.org/r/20210703151306.1922450-3-kw@linux.com
Link: https://lore.kernel.org/r/20210703151306.1922450-4-kw@linux.com
Link: https://lore.kernel.org/r/20210703151306.1922450-5-kw@linux.comSigned-off-by: NKrzysztof Wilczyński <kw@linux.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

Downstream Port Containment (PCIe r5.0, sec. 6.2.10) disables the link upon
an error and attempts to re-enable it when instructed by the DPC driver.

A slot which is both DPC- and hotplug-capable is currently powered off by
pciehp once DPC is triggered (due to the link change) and powered back up
on successful recovery.  That's undesirable, the slot should remain powered
so the hotplugged device remains bound to its driver.  DPC notifies the
driver of the error and of successful recovery in pcie_do_recovery() and
the driver may then restore the device to working state.

Moreover, Sinan points out that turning off slot power by pciehp may foil
recovery by DPC:  Power off/on is a cold reset concurrently to DPC's warm
reset.  Sathyanarayanan reports extended delays or failure in link
retraining by DPC if pciehp brings down the slot.

Fix by detecting whether a Link Down event is caused by DPC and awaiting
recovery if so.  On successful recovery, ignore both the Link Down and the
subsequent Link Up event.

Afterwards, check whether the link is down to detect surprise-removal or
another DPC event immediately after DPC recovery.  Ensure that the
corresponding DLLSC event is not ignored by synthesizing it and invoking
irq_wake_thread() to trigger a re-run of pciehp_ist().

The IRQ threads of the hotplug and DPC drivers, pciehp_ist() and
dpc_handler(), race against each other.  If pciehp is faster than DPC, it
will wait until DPC recovery completes.

Recovery consists of two steps:  The first step (waiting for link
disablement) is recognizable by pciehp through a set DPC Trigger Status
bit.  The second step (waiting for link retraining) is recognizable through
a newly introduced PCI_DPC_RECOVERING flag.

If DPC is faster than pciehp, neither of the two flags will be set and
pciehp may glean the recovery status from the new PCI_DPC_RECOVERED flag.
The flag is zero if DPC didn't occur at all, hence DLLSC events are not
ignored by default.

pciehp waits up to 4 seconds before assuming that DPC recovery failed and
bringing down the slot.  This timeout is not taken from the spec (it
doesn't mandate one) but based on a report from Yicong Yang that DPC may
take a bit more than 3 seconds on HiSilicon's Kunpeng platform.

The timeout is necessary because the DPC Trigger Status bit may never
clear:  On Root Ports which support RP Extensions for DPC, the DPC driver
polls the DPC RP Busy bit for up to 1 second before giving up on DPC
recovery.  Without the timeout, pciehp would then wait indefinitely for DPC
to complete.

This commit draws inspiration from previous attempts to synchronize DPC
with pciehp:

By Sinan Kaya, August 2018:
https://lore.kernel.org/linux-pci/20180818065126.77912-1-okaya@kernel.org/

By Ethan Zhao, October 2020:
https://lore.kernel.org/linux-pci/20201007113158.48933-1-haifeng.zhao@intel.com/

By Kuppuswamy Sathyanarayanan, March 2021:
https://lore.kernel.org/linux-pci/59cb30f5e5ac6d65427ceaadf1012b2ba8dbf66c.1615606143.git.sathyanarayanan.kuppuswamy@linux.intel.com/

Link: https://lore.kernel.org/r/0be565d97438fe2a6d57354b3aa4e8626952a00b.1619857124.git.lukas@wunner.deReported-by: NSinan Kaya <okaya@kernel.org>
Reported-by: NEthan Zhao <haifeng.zhao@intel.com>
Reported-by: NKuppuswamy Sathyanarayanan <sathyanarayanan.kuppuswamy@linux.intel.com>
Tested-by: NKuppuswamy Sathyanarayanan <sathyanarayanan.kuppuswamy@linux.intel.com>
Tested-by: NYicong Yang <yangyicong@hisilicon.com>
Signed-off-by: NLukas Wunner <lukas@wunner.de>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Reviewed-by: NKuppuswamy Sathyanarayanan <sathyanarayanan.kuppuswamy@linux.intel.com>
Cc: Dan Williams <dan.j.williams@intel.com>
Cc: Ashok Raj <ashok.raj@intel.com>
Cc: Keith Busch <kbusch@kernel.org>

pci_init_capabilities() is the only caller and doesn't use the return
value. So let's change the return type to void.

Link: https://lore.kernel.org/r/663ec440-8375-1459-ddb4-98ea76e75917@gmail.comSigned-off-by: NHeiner Kallweit <hkallweit1@gmail.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

The "label", "index", and "acpi_index" sysfs attributes show firmware label
information about the device.  If the ACPI Device Name _DSM is implemented
for the device, we have:

  label        Device name (optional, may be null)
  acpi_index   Instance number (unique under \_SB scope)

When there is no ACPI _DSM and SMBIOS provides an Onboard Devices structure
for the device, we have:

  label        Reference Designation, e.g., a silkscreen label
  index        Device Type Instance

Previously these attributes were dynamically created either by
pci_bus_add_device() or the pci_sysfs_init() initcall, but since they don't
need to be created or removed dynamically, we can use a static attribute so
the device model takes care of addition and removal automatically.

Convert "label", "index", and "acpi_index" to static attributes.

Presence of the ACPI _DSM (device_has_acpi_name()) determines whether the
ACPI information (label, acpi_index) or the SMBIOS information (label,
index) is visible.

[bhelgaas: commit log, split to separate patch, add "pci_dev_" prefix]
Suggested-by: NOliver O'Halloran <oohall@gmail.com>
Link: https://lore.kernel.org/r/20210416205856.3234481-6-kw@linux.comSigned-off-by: NKrzysztof Wilczyński <kw@linux.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

The "vpd" sysfs attribute allows access to Vital Product Data (VPD).
Previously it was dynamically created either by pci_bus_add_device() or the
pci_sysfs_init() initcall, but since it doesn't need to be created or
removed dynamically, we can use a static attribute so the device model
takes care of addition and removal automatically.

Convert "vpd" to a static attribute and use the .is_bin_visible() callback
to check whether the device supports VPD.

Remove pcie_vpd_create_sysfs_dev_files(),
pcie_vpd_remove_sysfs_dev_files(), pci_create_capabilities_sysfs(), and
pci_create_capabilities_sysfs(), which are no longer needed.

[bhelgaas: This is substantially the same as the earlier patch from Heiner
Kallweit <hkallweit1@gmail.com>.  I included Krzysztof's change here so all
the "convert to static attribute" changes are together.]

[bhelgaas: rename to vpd_read()/vpd_write() and pci_dev_vpd_attr_group]
Suggested-by: NOliver O'Halloran <oohall@gmail.com>
Based-on: https://lore.kernel.org/r/7703024f-8882-9eec-a122-599871728a89@gmail.comBased-on-patch-by: NHeiner Kallweit <hkallweit1@gmail.com>
Link: https://lore.kernel.org/r/20210416205856.3234481-5-kw@linux.comSigned-off-by: NKrzysztof Wilczyński <kw@linux.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

A typical cloud provider SR-IOV use case is to create many VFs for use by
guest VMs. The VFs may not be assigned to a VM until a customer requests a
VM of a certain size, e.g., number of CPUs. A VF may need MSI-X vectors
proportional to the number of CPUs in the VM, but there is no standard way
to change the number of MSI-X vectors supported by a VF.

Some Mellanox ConnectX devices support dynamic assignment of MSI-X vectors
to SR-IOV VFs. This can be done by the PF driver after VFs are enabled,
and it can be done without affecting VFs that are already in use. The
hardware supports a limited pool of MSI-X vectors that can be assigned to
the PF or to individual VFs.  This is device-specific behavior that
requires support in the PF driver.

Add a read-only "sriov_vf_total_msix" sysfs file for the PF and a writable
"sriov_vf_msix_count" file for each VF. Management software may use these
to learn how many MSI-X vectors are available and to dynamically assign
them to VFs before the VFs are passed through to a VM.

If the PF driver implements the ->sriov_get_vf_total_msix() callback,
"sriov_vf_total_msix" contains the total number of MSI-X vectors available
for distribution among VFs.

If no driver is bound to the VF, writing "N" to "sriov_vf_msix_count" uses
the PF driver ->sriov_set_msix_vec_count() callback to assign "N" MSI-X
vectors to the VF.  When a VF driver subsequently reads the MSI-X Message
Control register, it will see the new Table Size "N".

Link: https://lore.kernel.org/linux-pci/20210314124256.70253-2-leon@kernel.orgAcked-by: NBjorn Helgaas <bhelgaas@google.com>
Signed-off-by: NLeon Romanovsky <leonro@nvidia.com>

Compile-testing these drivers is currently broken. Enabling it causes a
couple of build failures though:

  drivers/pci/controller/pci-thunder-ecam.c:119:30: error: shift count >= width of type [-Werror,-Wshift-count-overflow]
  drivers/pci/controller/pci-thunder-pem.c:54:2: error: implicit declaration of function 'writeq' [-Werror,-Wimplicit-function-declaration]
  drivers/pci/controller/pci-thunder-pem.c:392:8: error: implicit declaration of function 'acpi_get_rc_resources' [-Werror,-Wimplicit-function-declaration]

Fix them with the obvious one-line changes.

Link: https://lore.kernel.org/r/20210308152501.2135937-2-arnd@kernel.orgSigned-off-by: NArnd Bergmann <arnd@arndb.de>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Reviewed-by: NKuppuswamy Sathyanarayanan <sathyanarayanan.kuppuswamy@linux.intel.com>
Reviewed-by: NRobert Richter <rric@kernel.org>

This reverts commit 4257f7e0.

Kenneth reported that after 4257f7e0, he sees a torrent of disk I/O
errors on his NVMe device after suspend/resume until a reboot.

Link: https://lore.kernel.org/linux-pci/20201228040513.GA611645@bjorn-Precision-5520/Reported-by: NKenneth R. Crudup <kenny@panix.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

Export pci_rebar_get_possible_sizes() for use by modular drivers.
Signed-off-by: NDarren Salt <devspam@moreofthesa.me.uk>
Signed-off-by: NNirmoy Das <nirmoy.das@amd.com>
Signed-off-by: NChristian König <christian.koenig@amd.com>
Acked-by: NBjorn Helgaas <bhelgaas@google.com>
Link: https://patchwork.kernel.org/project/dri-devel/patch/20210107175017.15893-2-nirmoy.das@amd.com

There are systems (for example, Intel based mobile platforms since Coffee
Lake) where the power drawn while suspended can be significantly reduced by
disabling Precision Time Measurement (PTM) on PCIe root ports as this
allows the port to enter a lower-power PM state and the SoC to reach a
lower-power idle state. To save this power, disable the PTM feature on root
ports during pci_prepare_to_sleep() and pci_finish_runtime_suspend().  The
feature will be returned to its previous state during restore and error
recovery.
Suggested-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>
Bugzilla: https://bugzilla.kernel.org/show_bug.cgi?id=209361
Link: https://lore.kernel.org/r/20201207223951.19667-2-david.e.box@linux.intel.comReported-by: NLen Brown <len.brown@intel.com>
Signed-off-by: NDavid E. Box <david.e.box@linux.intel.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

The PCI subsystem does not currently save and restore the configuration
space for the Precision Time Measurement (PTM) Extended Capability leading
to the possibility of the feature returning disabled on S3 resume.  This
has been observed on Intel Coffee Lake desktops. Add save/restore of the
PTM control register. This saves the PTM Enable, Root Select, and Effective
Granularity bits.
Suggested-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>
Link: https://lore.kernel.org/r/20201207223951.19667-1-david.e.box@linux.intel.comSigned-off-by: NDavid E. Box <david.e.box@linux.intel.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

Root Complex Event Collectors (RCEC) appear as peers to Root Ports and also
have the AER capability. In addition, actions need to be taken for
associated RCiEPs. In such cases the RCECs will need to be walked in order
to find and act upon their respective RCiEPs.

Extend the existing ability to link the RCECs with a walking function
pcie_walk_rcec(). Add RCEC support to the current AER service driver and
attach the AER service driver to the RCEC device.
Co-developed-by: NQiuxu Zhuo <qiuxu.zhuo@intel.com>
Link: https://lore.kernel.org/r/20201121001036.8560-14-sean.v.kelley@intel.com
Tested-by: Jonathan Cameron <Jonathan.Cameron@huawei.com> # non-native/no RCEC
Signed-off-by: NQiuxu Zhuo <qiuxu.zhuo@intel.com>
Signed-off-by: NSean V Kelley <sean.v.kelley@intel.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Reviewed-by: NJonathan Cameron <Jonathan.Cameron@huawei.com>
Reviewed-by: NKuppuswamy Sathyanarayanan <sathyanarayanan.kuppuswamy@linux.intel.com>

A Root Complex Event Collector terminates error and PME messages from
associated RCiEPs.

Use the RCEC Endpoint Association Extended Capability to identify
associated RCiEPs. Link the associated RCiEPs as the RCECs are enumerated.
Co-developed-by: NQiuxu Zhuo <qiuxu.zhuo@intel.com>
Link: https://lore.kernel.org/r/20201121001036.8560-12-sean.v.kelley@intel.com
Tested-by: Jonathan Cameron <Jonathan.Cameron@huawei.com> # non-native/no RCEC
Signed-off-by: NQiuxu Zhuo <qiuxu.zhuo@intel.com>
Signed-off-by: NSean V Kelley <sean.v.kelley@intel.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Reviewed-by: NJonathan Cameron <Jonathan.Cameron@huawei.com>

pci_msi_set_enable() and pci_msix_clear_and_set_ctrl() are only used from
msi.c, so move them from drivers/pci/pci.h to msi.c.  No functional change
intended.

Link: https://lore.kernel.org/r/20201203185110.1583077-3-helgaas@kernel.orgSigned-off-by: NBjorn Helgaas <bhelgaas@google.com>
Reviewed-by: NThierry Reding <treding@nvidia.com>

Move pci_msi_setup_pci_dev(), which disables MSI and MSI-X interrupts, from
probe.c to msi.c so it's with all the other MSI code and more consistent
with other capability initialization.  This means we must compile msi.c
always, even without CONFIG_PCI_MSI, so wrap the rest of msi.c in an #ifdef
and adjust the Makefile accordingly.  No functional change intended.

Link: https://lore.kernel.org/r/20201203185110.1583077-2-helgaas@kernel.orgSigned-off-by: NBjorn Helgaas <bhelgaas@google.com>
Reviewed-by: NThierry Reding <treding@nvidia.com>

reset_link() appears to be misnamed.  The point is to reset any devices
below a given bridge, so rename it to reset_subordinates() to make it clear
that we are passing a bridge with the intent to reset the devices below it.

Link: https://lore.kernel.org/r/20201121001036.8560-5-sean.v.kelley@intel.com
Tested-by: Jonathan Cameron <Jonathan.Cameron@huawei.com> # non-native/no RCEC
Signed-off-by: NSean V Kelley <sean.v.kelley@intel.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Reviewed-by: NKuppuswamy Sathyanarayanan <sathyanarayanan.kuppuswamy@linux.intel.com>
Acked-by: NJonathan Cameron <Jonathan.Cameron@huawei.com>

Extend support for Root Complex Event Collectors by decoding and caching
the RCEC Endpoint Association Extended Capabilities when enumerating. Use
that cached information for later error source reporting. See PCIe r5.0,
sec 7.9.10.
Co-developed-by: NQiuxu Zhuo <qiuxu.zhuo@intel.com>
Link: https://lore.kernel.org/r/20201121001036.8560-4-sean.v.kelley@intel.com
Tested-by: Jonathan Cameron <Jonathan.Cameron@huawei.com> # non-native/no RCEC
Signed-off-by: NQiuxu Zhuo <qiuxu.zhuo@intel.com>
Signed-off-by: NSean V Kelley <sean.v.kelley@intel.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Reviewed-by: NJonathan Cameron <Jonathan.Cameron@huawei.com>

PCIe r6.0, sec 7.5.3.18, defines a new 64.0 GT/s bit in the Supported Link
Speeds Vector of Link Capabilities 2.

This patch does not affect the speed of the link, which should be
negotiated automatically by the hardware; it only adds decoding when
showing the speed to the user.

Decode this new speed.  Previously, reading the speed of a link operating
at this speed showed "Unknown speed" instead of "64.0 GT/s".

Link: https://lore.kernel.org/r/aaaab33fe18975e123a84aebce2adb85f44e2bbe.1605739760.git.gustavo.pimentel@synopsys.comSigned-off-by: NGustavo Pimentel <gustavo.pimentel@synopsys.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Reviewed-by: NKrzysztof Wilczyński <kw@linux.com>

Previously ASPM L1 Substates control registers (CTL1 and CTL2) weren't
saved and restored during suspend/resume leading to L1 Substates
configuration being lost post-resume.

Save the L1 Substates control registers so that the configuration is
retained post-resume.

Link: https://lore.kernel.org/r/20201024190442.871-1-vidyas@nvidia.comSigned-off-by: NVidya Sagar <vidyas@nvidia.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

This reverts commit 7e24bc34.

7e24bc34 was based on PCIe r5.0, sec 5.9, which claims we need a 200 ms
delay when transitioning to or from D2.  However, sec 5.3.1.3 states the
delay as 200 μs (microseconds), as does the table in PCIe r4.0, sec 5.9.1.

This looks like a typo in the r5.0 spec, so revert back to a 200 μs delay
instead of a 200 ms delay.

Fixes: 7e24bc34 ("PCI/PM: Apply D2 delay as milliseconds, not microseconds")
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Reviewed-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

476e7fae ("PCI PM: Do not wait for buses in B2 or B3 during resume")
removed the last use of PCI_PM_BUS_WAIT.  Remove the definition as well.
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Reviewed-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

PCI devices support two variants of the D3 power state: D3hot (main power
present) D3cold (main power removed).  Previously struct pci_dev contained:

  unsigned int    d3_delay;       /* D3->D0 transition time in ms */
  unsigned int    d3cold_delay;   /* D3cold->D0 transition time in ms */

"d3_delay" refers specifically to the D3hot state.  Rename it to
"d3hot_delay" to avoid ambiguity and align with the ACPI "_DSM for
Specifying Device Readiness Durations" in the PCI Firmware spec r3.2,
sec 4.6.9.

There is no change to the functionality.

Link: https://lore.kernel.org/r/20200730210848.1578826-1-kw@linux.comSigned-off-by: NKrzysztof Wilczyński <kw@linux.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

Now that pci_parse_request_of_pci_ranges() callers just setup
pci_host_bridge.windows and dma_ranges directly and don't need the bus
range returned, we can just initialize them when allocating the
pci_host_bridge struct.

With this, pci_parse_request_of_pci_ranges() becomes a static function.

Link: https://lore.kernel.org/r/20200722022514.1283916-19-robh@kernel.orgSigned-off-by: NRob Herring <robh@kernel.org>
Signed-off-by: NLorenzo Pieralisi <lorenzo.pieralisi@arm.com>
Acked-by: NBjorn Helgaas <bhelgaas@google.com>
Cc: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
Cc: Bjorn Helgaas <bhelgaas@google.com>

pci_aer_clear_device_status() clears the error bits in the PCIe Device
Status Register (PCI_EXP_DEVSTA).  Every PCIe device has this register,
regardless of whether it supports AER.

Rename pci_aer_clear_device_status() to pcie_clear_device_status() to make
clear that it is PCIe-specific but not AER-specific.  Move it to
drivers/pci/pci.c, again since it's not AER-specific.  No functional change
intended.

Link: https://lore.kernel.org/r/20200717195619.766662-1-helgaas@kernel.orgSigned-off-by: NBjorn Helgaas <bhelgaas@google.com>

Currently the ACS capability is being looked up at a number of places. Read
and store it once at enumeration so that it can be used by all later.  No
functional change intended.

Link: https://lore.kernel.org/r/20200707224604.3737893-2-rajatja@google.comSigned-off-by: NRajat Jain <rajatja@google.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

The method struct pci_error_handlers.error_detected() is defined and
documented as taking an 'enum pci_channel_state' for the second argument,
but most drivers use 'pci_channel_state_t' instead.

This 'pci_channel_state_t' is not a typedef for the enum but a typedef for
a bitwise type in order to have better/stricter typechecking.

Consolidate everything by using 'pci_channel_state_t' in the method's
definition, in the related helpers and in the drivers.

Enforce use of 'pci_channel_state_t' by replacing 'enum pci_channel_state'
with an anonymous 'enum'.

Note: Currently, from a typechecking point of view this patch changes
nothing because only the constants defined by the enum are bitwise, not the
enum itself (sparse doesn't have the notion of 'bitwise enum'). This may
change in some not too far future, hence the patch.

[bhelgaas: squash in
  https://lore.kernel.org/r/20200702162651.49526-3-luc.vanoostenryck@gmail.com
  https://lore.kernel.org/r/20200702162651.49526-4-luc.vanoostenryck@gmail.com]
Link: https://lore.kernel.org/r/20200702162651.49526-2-luc.vanoostenryck@gmail.comSigned-off-by: NLuc Van Oostenryck <luc.vanoostenryck@gmail.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

The AER interfaces to clear error status registers were a confusing mess:

  - pci_cleanup_aer_uncorrect_error_status() cleared non-fatal errors
    from the Uncorrectable Error Status register.

  - pci_aer_clear_fatal_status() cleared fatal errors from the
    Uncorrectable Error Status register.

  - pci_cleanup_aer_error_status_regs() cleared the Root Error Status
    register (for Root Ports), the Uncorrectable Error Status register,
    and the Correctable Error Status register.

Rename them to make them consistent:

  From                                     To
  ---------------------------------------- -------------------------------
  pci_cleanup_aer_uncorrect_error_status() pci_aer_clear_nonfatal_status()
  pci_aer_clear_fatal_status()             pci_aer_clear_fatal_status()
  pci_cleanup_aer_error_status_regs()      pci_aer_clear_status()

Since pci_cleanup_aer_error_status_regs() (renamed to
pci_aer_clear_status()) is only used within drivers/pci/, move the
declaration from <linux/aer.h> to drivers/pci/pci.h.

[bhelgaas: commit log, add renames]
Link: https://lore.kernel.org/r/d1310a75dc3d28f7e8da4e99c45fbd3e60fe238e.1585000084.git.sathyanarayanan.kuppuswamy@linux.intel.comSigned-off-by: NKuppuswamy Sathyanarayanan <sathyanarayanan.kuppuswamy@linux.intel.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

If firmware controls DPC, it is generally responsible for managing the DPC
capability and events, and the OS should not access the DPC capability.

However, if firmware controls DPC and both the OS and the platform support
Error Disconnect Recover (EDR) notifications, the OS EDR notify handler is
responsible for recovery, and the notify handler may read/write the DPC
capability until it clears the DPC Trigger Status bit.  See [1], sec 4.5.1,
table 4-6.

Expose some DPC error handling functions so they can be used by the EDR
notify handler.

[1] Downstream Port Containment Related Enhancements ECN, Jan 28, 2019,
    affecting PCI Firmware Specification, Rev. 3.2
    https://members.pcisig.com/wg/PCI-SIG/document/12888

Link: https://lore.kernel.org/r/e9000bb15b3a4293e81d98bb29ead7c84a6393c9.1585000084.git.sathyanarayanan.kuppuswamy@linux.intel.comSigned-off-by: NKuppuswamy Sathyanarayanan <sathyanarayanan.kuppuswamy@linux.intel.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

Per the SFI _OSC and DPC Updates ECN [1] implementation note flowchart, the
OS seems to be expected to clear AER status even if it doesn't have
ownership of the AER capability.  Unlike the DPC capability, where a DPC
ECN [2] specifies a window when the OS is allowed to access DPC registers
even if it doesn't have ownership, there is no clear model for AER.

Add pci_aer_raw_clear_status() to clear the AER error status registers
unconditionally.  This is intended for use only by the EDR path (see [2]).

[1] System Firmware Intermediary (SFI) _OSC and DPC Updates ECN, Feb 24,
    2020, affecting PCI Firmware Specification, Rev. 3.2
    https://members.pcisig.com/wg/PCI-SIG/document/14076
[2] Downstream Port Containment Related Enhancements ECN, Jan 28, 2019,
    affecting PCI Firmware Specification, Rev. 3.2
    https://members.pcisig.com/wg/PCI-SIG/document/12888

[bhelgaas: changelog]
Link: https://lore.kernel.org/r/c19ad28f3633cce67448609e89a75635da0da07d.1585000084.git.sathyanarayanan.kuppuswamy@linux.intel.comSigned-off-by: NKuppuswamy Sathyanarayanan <sathyanarayanan.kuppuswamy@linux.intel.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

Since Error Disconnect Recover needs to use DPC error handling routines
even if the OS doesn't have control of DPC, move the initalization and
caching of DPC capabilities from the DPC driver to pci_init_capabilities().

Link: https://lore.kernel.org/r/5888380657c8b9551675b5dbd48e370e4fd2703d.1585000084.git.sathyanarayanan.kuppuswamy@linux.intel.comSigned-off-by: NKuppuswamy Sathyanarayanan <sathyanarayanan.kuppuswamy@linux.intel.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

As per the DPC Enhancements ECN [1], sec 4.5.1, table 4-4, if the OS
supports Error Disconnect Recover (EDR), it must invalidate the software
state associated with child devices of the port without attempting to
access the child device hardware. In addition, if the OS supports DPC, it
must attempt to recover the child devices if the port implements the DPC
Capability. If the OS continues operation, the OS must inform the firmware
of the status of the recovery operation via the _OST method.

Return the result of pcie_do_recovery() so we can report it to firmware via
_OST.

[1] Downstream Port Containment Related Enhancements ECN, Jan 28, 2019,
    affecting PCI Firmware Specification, Rev. 3.2
    https://members.pcisig.com/wg/PCI-SIG/document/12888

Link: https://lore.kernel.org/r/eb60ec89448769349c6722954ffbf2de163155b5.1585000084.git.sathyanarayanan.kuppuswamy@linux.intel.comSigned-off-by: NKuppuswamy Sathyanarayanan <sathyanarayanan.kuppuswamy@linux.intel.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

Previously we passed the PCIe service type parameter to pcie_do_recovery(),
where reset_link() looked up the underlying pci_port_service_driver and its
.reset_link() function pointer. Instead of using this roundabout way, we
can just pass the driver-specific .reset_link() callback function when
calling pcie_do_recovery() function.

This allows us to call pcie_do_recovery() from code that is not a PCIe port
service driver, e.g., Error Disconnect Recover (EDR) support.

Remove pcie_port_find_service() and pcie_port_service_driver.reset_link
since they are now unused.

Link: https://lore.kernel.org/r/60e02b87b526cdf2930400059d98704bf0a147d1.1585000084.git.sathyanarayanan.kuppuswamy@linux.intel.comSigned-off-by: NKuppuswamy Sathyanarayanan <sathyanarayanan.kuppuswamy@linux.intel.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

Add PCIE_LNKCAP2_SLS2SPEED macro for transforming raw Link Capabilities 2
values to the pci_bus_speed. This is next to PCIE_SPEED2MBS_ENC() to make
it easier to update both places when adding support for new speeds.

Link: https://lore.kernel.org/r/1581937984-40353-10-git-send-email-yangyicong@hisilicon.comSigned-off-by: NYicong Yang <yangyicong@hisilicon.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

Previously some PCI speed strings came from pci_speed_string(), some came
from the PCIe-specific PCIE_SPEED2STR(), and some came from a PCIe-specific
switch statement.  These methods were inconsistent:

  pci_speed_string()     PCIE_SPEED2STR()     switch
  ------------------     ----------------     ------
  33 MHz PCI
  ...
  2.5 GT/s PCIe          2.5 GT/s             2.5 GT/s
  5.0 GT/s PCIe          5 GT/s               5 GT/s
  8.0 GT/s PCIe          8 GT/s               8 GT/s
  16.0 GT/s PCIe         16 GT/s              16 GT/s
  32.0 GT/s PCIe         32 GT/s              32 GT/s

Standardize on pci_speed_string() as the single source of these strings.

Note that this adds ".0" and "PCIe" to some messages, including sysfs
"max_link_speed" files, a brcmstb "link up" message, and the link status
dmesg logging, e.g.,

  nvme 0000:01:00.0: 16.000 Gb/s available PCIe bandwidth, limited by 5.0 GT/s PCIe x4 link at 0000:00:01.1 (capable of 31.504 Gb/s with 8.0 GT/s PCIe x4 link)

I think it's better to standardize on a single version of the speed text.
Previously we had strings like this:

  /sys/bus/pci/slots/0/cur_bus_speed: 8.0 GT/s PCIe
  /sys/bus/pci/slots/0/max_bus_speed: 8.0 GT/s PCIe
  /sys/devices/pci0000:00/0000:00:1c.0/current_link_speed: 8 GT/s
  /sys/devices/pci0000:00/0000:00:1c.0/max_link_speed: 8 GT/s

This changes the latter two to match the slots files:

  /sys/devices/pci0000:00/0000:00:1c.0/current_link_speed: 8.0 GT/s PCIe
  /sys/devices/pci0000:00/0000:00:1c.0/max_link_speed: 8.0 GT/s PCIe

Based-on-patch by: Yicong Yang <yangyicong@hisilicon.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

Add pci_speed_string() to return a text description of the supplied bus or
link speed.  The slot code previously used the private
pci_bus_speed_strings[] array for this purpose, but adding this interface
will enable us to consolidate similar code elsewhere.

Export pcie_link_speed[] and pci_speed_string() so they can be used by
modules.
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

Link speed 32.0 GT/s is supported in PCIe r5.0. Add this speed to
PCIE_SPEED2STR() and PCIE_SPEED2MBS_ENC() to correctly decode it.

This is complementary to de76cda2 ("PCI: Decode PCIe 32 GT/s link
speed").

Link: https://lore.kernel.org/r/1581937984-40353-2-git-send-email-yangyicong@hisilicon.comSigned-off-by: NYicong Yang <yangyicong@hisilicon.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

The number of possible devfns is 256, but pci_add_dma_alias() allocated a
bitmap of size 255.  Fix this off-by-one error.

This fixes commits 338c3149 ("PCI: Add support for multiple DMA
aliases") and c6635792 ("PCI: Allocate dma_alias_mask with
bitmap_zalloc()"), but I doubt it was possible to see a problem because
it takes 4 64-bit longs (or 8 32-bit longs) to hold 255 bits, and
bitmap_zalloc() doesn't save the 255-bit size anywhere.

[bhelgaas: commit log, move #define to drivers/pci/pci.h, include loop
limit fix from Qian Cai:
https://lore.kernel.org/r/20191218170004.5297-1-cai@lca.pw]
Signed-off-by: NJames Sewart <jamessewart@arista.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Reviewed-by: NLogan Gunthorpe <logang@deltatee.com>

Previously, CONFIG_PCIEASPM_DEBUG enabled "link_state" and "clk_ctl" sysfs
files that controlled ASPM.  We believe these files were rarely if ever
used.

We recently added sysfs ASPM controls that are always present, so the debug
code is no longer needed.  Removing this debug code has been discussed for
quite some time, see e.g. [0].

Remove PCIEASPM_DEBUG and the related code.

[0] https://lore.kernel.org/lkml/20180727202619.GD173328@bhelgaas-glaptop.roam.corp.google.com/
Link: https://lore.kernel.org/r/ec935d8e-c084-3938-f1d1-748617596b25@gmail.comSigned-off-by: NHeiner Kallweit <hkallweit1@gmail.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

Add sysfs attributes to Endpoints and other Upstream Ports to control ASPM,
Clock PM, and L1 PM Substates.  The new attributes are:

  /sys/devices/pci*/.../link/clkpm
  /sys/devices/pci*/.../link/l0s_aspm
  /sys/devices/pci*/.../link/l1_aspm
  /sys/devices/pci*/.../link/l1_1_aspm
  /sys/devices/pci*/.../link/l1_2_aspm
  /sys/devices/pci*/.../link/l1_1_pcipm
  /sys/devices/pci*/.../link/l1_2_pcipm

An attribute is only visible if both ends of the Link leading to the device
support the state.  Writing y/1/on to the file enables the state; n/0/off
disables it.

These attributes can be used to tune the power/performance tradeoff for
individual devices.

[bhelgaas: commit log, rename directory to "link"]
Link: https://lore.kernel.org/r/b1c83f8a-9bf6-eac5-82d0-cf5b90128fbf@gmail.comSigned-off-by: NHeiner Kallweit <hkallweit1@gmail.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>