Per the PCI Firmware spec, r3.0, sec 4.5.1, on ACPI systems, the OS must
not use AER unless _OSC is present and _OSC grants AER control to the OS.
The aerdriver.forceload kernel parameter was a way to enable Linux AER
support on ACPI systems that lack _OSC or fail to grant control the the OS.

Enabling Linux AER support when the firmware doesn't want us to is a recipe
for problems, e.g., the firmware might be handling AER itself.

Remove the aerdriver.forceload kernel parameter and related supporting
code.
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

The aerdriver.nosourceid kernel parameter was intended for working around
broken chipsets don't supply the source ID for AER events.  We recently
added PCI_BUS_FLAGS_NO_AERSID, which can be set by quirks for the same
purpose.

Remove the aerdriver.nosourceid kernel parameter.  For anything other than
debugging, asking users to find and use kernel parameters is a poor user
experience.  Instead, we should add PCI_BUS_FLAGS_NO_AERSID quirks for any
hardware that needs it.
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

Allow root port buses to choose to skip source id matching when finding the
faulting device.  Certain root port devices may return an incorrect source
ID and recommend to scan child device registers for AER notifications.
Signed-off-by: NJon Derrick <jonathan.derrick@intel.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

Change the Downstream Port Containment config type from tristate to bool.

The driver doesn't automatically load based on any rules, so it needs to be
built-in in order to bind to devices it needs to drive.
Signed-off-by: NKeith Busch <keith.busch@intel.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

PCIe port type values are not flags, so OR'ing them is not correct.
Previously the result was equivalent to PCIe Downstream Ports, so we were
missing binding to DPC-capable Root Ports.

Change the type to 'any' so we can bind to both port types.  While this
will cause the code to check Upstream Ports, the driver won't claim them
since they are not DPC-capable.
Reported-by: NAlexander Antonov <alexanderx.v.antonov@intel.com>
Signed-off-by: NKeith Busch <keith.busch@intel.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
CC: Mika Westerberg <mika.westerberg@linux.intel.com>

Remove unnecessary spaces before tabs.
Signed-off-by: NMika Westerberg <mika.westerberg@linux.intel.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Acked-by: NKeith Busch <keith.busch@intel.com>

Use the device resource management (devm) interfaces so we don't need to
explicitly release resources on failure paths or when the driver is
removed.
Signed-off-by: NMika Westerberg <mika.westerberg@linux.intel.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Acked-by: NKeith Busch <keith.busch@intel.com>

Add back runtime PM support for PCIe ports that was removed by
fe9a743a ("PCI/PM: Drop unused runtime PM support code for PCIe
ports").

We cannot enable it automatically for all ports since there have been
problems previously [1].  In summary suspended PCIe ports were not able
to deal with ACPI-based hotplug reliably.  One reason why this might happen
is the fact that when a PCIe port is powered down, config space access to
the devices behind the port is not possible.  If the BIOS hotplug SMI
handler assumes the port is always in D0 it will not be able to find the
hotplugged devices.  To be on the safe side only enable runtime PM if the
port does not claim to support hotplug.

For PCIe ports not using hotplug, we enable and allow runtime PM
automatically.  Since 'bridge_d3' can be changed any time we check this in
driver ->runtime_idle() and ->runtime_suspend() and only allow runtime
suspend if the flag is still set.  Use autosuspend with default of 100ms
idle time to prevent the port from repeatedly suspending and resuming on
continuous configuration space access of devices behind the port.

The actual power transition to D3 and back is handled in the PCI core.

Idea to automatically unblock (allow) runtime PM for PCIe ports came from
Dave Airlie.

[1] https://bugzilla.kernel.org/show_bug.cgi?id=53811

This includes a fix for lockdep issue reported by Valdis Kletnieks.
Tested-by: NLukas Wunner <lukas@wunner.de>
Signed-off-by: NLukas Wunner <lukas@wunner.de>
Signed-off-by: NMika Westerberg <mika.westerberg@linux.intel.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Acked-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

The PCI core skips bridges and ports when the system is suspended.  The PCI
core checks return value of pci_has_subordinate() in pci_pm_suspend_noirq()
to skip all devices where it is non-zero (which means PCI bridges and PCIe
ports).

Since PCIe ports are never suspended in the first place, there is no need
to set d3cold_allowed for them.
Tested-by: NLukas Wunner <lukas@wunner.de>
Signed-off-by: NMika Westerberg <mika.westerberg@linux.intel.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Acked-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

Without supporting clock PM capable, if we want to disable clkpm, we don't
need this extra check as it must already be zero for the enable argument.
And it's the same for enabling clkpm here.  So let's remove this check.
Signed-off-by: NShawn Lin <shawn.lin@rock-chips.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

We cache the PCI_EXP_SLTCAP_HPC bit in pci_dev->is_hotplug_bridge on device
probe, so there's no need to read it again on allocation of port service
devices.

No functional change intended.
Signed-off-by: NLukas Wunner <lukas@wunner.de>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

Add driver for the PCI Express Downstream Port Containment extended
capability.  DPC is an optional capability to contain uncorrectable errors
below a port.

For more information on DPC, please see PCI Express Base Specification
Revision 4, section 7.31, or view the PCI-SIG DPC ECN here:

  https://pcisig.com/sites/default/files/specification_documents/ECN_DPC_2012-02-09_finalized.pdf

When a DPC event is triggered, the hardware disables downstream links, so
the DPC driver schedules removal for all devices below this port.  This may
happen concurrently with a PCIe hotplug driver if enabled.  When all
downstream devices are removed and the link state transitions to disabled,
the DPC driver clears the DPC status and interrupt bits so the link may
retrain for a newly connected device.

[bhelgaas: clear (not set) DPC_CTL bits on remove, whitespace cleanup]
Signed-off-by: NKeith Busch <keith.busch@intel.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Cc: Lukas Wunner <lukas@wunner.de>

Add the Downstream Port Containment (PCIE_PORT_SERVICE_DPC) portdrv service
type, available if the device has the DPC extended capability.

[bhelgaas: split to separate patch, changelog]
Signed-off-by: NKeith Busch <keith.busch@intel.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

The names of port service devices previously used one nibble to encode the
port type and another nibble to encode the service type.  We're about to
add a fifth service type, so change device names to use one *byte* to
encode the service type.

For example, a hotplug port service on a downstream bridge was previously
called "pcie24" and is now called "pcie204".  The "2" encodes the device
type (PCI_EXP_TYPE_DOWNSTREAM - 4), and the "4" (now "04") encodes the
service (PCIE_PORT_SERVICE_HP).

Based on Lukas Wunner's patch:
https://github.com/l1k/linux/commit/b688d6e4873ab082e5916b1a993bc1d38c6f4178

[bhelgaas: split to separate patch, expand changelog]
Based-on-patch-by: NLukas Wunner <lukas@wunner.de>
Signed-off-by: NKeith Busch <keith.busch@intel.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

Now that pcie_port_acpi_setup() always returns 0, make it and its callers
void functions and stop checking the return values.

[bhelgaas: changelog]
Signed-off-by: NJon Derrick <jonathan.derrick@intel.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

Host bridges we discover via ACPI, i.e., PNP0A03 and PNP0A08 devices, may
have an _OSC method by which the OS can ask the platform for control of
PCIe features like native hotplug, power management events, AER, etc.

Previously, if we found a bridge without an ACPI device, we assumed we did
not have permission to use any of these PCIe features.  That seems
unreasonably restrictive.

If we find no ACPI device, assume we can take control of all PCIe features.

The Intel Volume Management Device (VMD) is one such bridge with no ACPI
device.  Prior to this change, users had to boot with "pcie_ports=native"
to get hotplug and other services to work below the VMD Root Port.

[bhelgaas: changelog]
Suggested-by: NBjorn Helgaas <helgaas@kernel.org>
Signed-off-by: NJon Derrick <jonathan.derrick@intel.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

Clean up style issues in drivers/pci/pcie/Kconfig, in particular all
indentation is now done using tabs, not spaces, and the definition of
PCIEASPM_DEBUG is now separated from the definition of PCIEASPM with a
newline.
Signed-off-by: NAndreas Ziegler <andreas.ziegler@fau.de>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

Log successful error injections so that injected errors can be
differentiated from real errors.
Suggested-by: NBjorn Helgaas <bhelgaas@google.com>
Signed-off-by: NJean Delvare <jdelvare@suse.de>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
CC: Borislav Petkov <bp@suse.de>

The aer_inject driver is very quiet.  In most cases, it merely returns an
error code to user-space, leaving the user with little clue about the
actual reason for the failure.

So, log error messages for 4 of the most frequent causes of failure:
* Can't find the root port of the specified device.
* Device doesn't support AER.
* Root port doesn't support AER.
* AER device not found.

This gives the user a chance to understand why aer-inject failed.

Based on a preliminary patch by Thomas Renninger.
Signed-off-by: NJean Delvare <jdelvare@suse.de>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
CC: Borislav Petkov <bp@suse.de>
CC: Thomas Renninger <trenn@suse.de>

dev_warn() is better than printk(LOG_WARNING...) as it records which device
the message relates to.  Also add a prefix "aer_inject:" to help
differentiate real errors from injected errors.
Signed-off-by: NJean Delvare <jdelvare@suse.de>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
CC: Borislav Petkov <bp@suse.de>

EPERM means "Operation not permitted", which doesn't reflect the lack of
support for AER.  EPROTONOSUPPORT (Protocol not supported) is a better
choice of error code if the device or its root port lack support for AER.

Likewise, EINVAL means "Invalid argument", which is not suitable for cases
where the AER error device is missing or unusable.  ENODEV and
EPROTONOSUPPORT, respectively, fit better.
Suggested-by: NBorislav Petkov <bp@suse.de>
Signed-off-by: NJean Delvare <jdelvare@suse.de>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
CC: Borislav Petkov <bp@suse.de>
CC: Prarit Bhargava <prarit@redhat.com>

Previously we had this:

  if (wakeup)
    ret = enable_irq_wake(...);
  if (!wakeup || ret)
    ...

"ret" is only evaluated when "wakeup" is true, and it is always initialized
in that case, but gcc isn't smart enough to figure that out and warns:

  drivers/pci/pcie/pme.c:414:14: warning: 'ret' may be used uninitialized in this function [-Wmaybe-uninitialized]

Restructure the code slightly to make it easier for gcc (and maybe for
humans as well).
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Acked-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com

We've already looked up srv->port a few lines earlier, and there's no need
to do it again.  Remove the redundant lookup.
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Acked-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com

Use list_first_entry_or_null() instead of list_empty() + list_entry() to
simplify the code.
Signed-off-by: NGeliang Tang <geliangtang@163.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

The aer_inject module intercepts config space accesses by replacing the
bus->ops pointer.  If it forwards accesses to the original pci_ops, and
those original ops use bus->ops, they see the aer_pci_ops instead of their
own pci_ops, which can cause a crash.

For example, pci_generic_config_read() uses the bus->ops->map_bus pointer.
If bus->ops is set to aer_pci_ops, which doesn't supply .map_bus,
pci_generic_config_read() will dereference an invalid pointer and cause a
crash.

Temporarily restore the original bus->ops pointer while calling ops->read()
or ops->write().  Callers of these functions already hold pci_lock, which
prevents other users of bus->ops until we're finished.

[bhelgaas: changelog]
Signed-off-by: NDavid Daney <david.daney@cavium.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

Rename

  pci_ops_aer     to aer_inj_pci_ops
  pci_read_aer()  to aer_inj_read_config()
  pci_write_aer() to aer_inj_write_config()

This is more conventional and more informative.  No functional change.
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

A Root Port's AER structure (rpc) contains a queue of events.  aer_irq()
enqueues AER status information and schedules aer_isr() to dequeue and
process it.  When we remove a device, aer_remove() waits for the queue to
be empty, then frees the rpc struct.

But aer_isr() references the rpc struct after dequeueing and possibly
emptying the queue, which can cause a use-after-free error as in the
following scenario with two threads, aer_isr() on the left and a
concurrent aer_remove() on the right:

  Thread A                      Thread B
  --------                      --------
  aer_irq():
    rpc->prod_idx++
                                aer_remove():
                                  wait_event(rpc->prod_idx == rpc->cons_idx)
                                  # now blocked until queue becomes empty
  aer_isr():                      # ...
    rpc->cons_idx++               # unblocked because queue is now empty
    ...                           kfree(rpc)
    mutex_unlock(&rpc->rpc_mutex)

To prevent this problem, use flush_work() to wait until the last scheduled
instance of aer_isr() has completed before freeing the rpc struct in
aer_remove().

I reproduced this use-after-free by flashing a device FPGA and
re-enumerating the bus to find the new device.  With SLUB debug, this
crashes with 0x6b bytes (POISON_FREE, the use-after-free magic number) in
GPR25:

  pcieport 0000:00:00.0: AER: Multiple Corrected error received: id=0000
  Unable to handle kernel paging request for data at address 0x27ef9e3e
  Workqueue: events aer_isr
  GPR24: dd6aa000 6b6b6b6b 605f8378 605f8360 d99b12c0 604fc674 606b1704 d99b12c0
  NIP [602f5328] pci_walk_bus+0xd4/0x104

[bhelgaas: changelog, stable tag]
Signed-off-by: NSebastian Andrzej Siewior <bigeasy@linutronix.de>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
CC: stable@vger.kernel.org

The Intel Volume Management Device (VMD) supports 32-bit domain numbers.
To accommodate this, use u32 instead of u16 to store domain numbers.

[bhelgaas: changelog]
Signed-off-by: NKeith Busch <keith.busch@intel.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

Bit 7 of the "Header Type" register indicates a multi-function device when
set.  Bits 0-6 contain encoded values, where 0x1 indicates a PCI-PCI
bridge.  It is incorrect to test this as though it were a mask.

For example, while the PCI 3.0 spec only defines values 0x0, 0x1, and 0x2,
it's conceivable that a future spec could define 0x3 to mean something
else; then tests for "(hdr_type & 0x7f) & PCI_HEADER_TYPE_BRIDGE" would
incorrectly succeed for this new 0x3 header type.

Test bits 0-6 of the Header Type for equality with PCI_HEADER_TYPE_BRIDGE.
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

If CONFIG_PCIEASPM_DEBUG is set, then PCI devices have a link_state
attribute.  Reading that attribute shows the state as a bit mask: 1
means L0S upstream, 2 means L0S downstream, and 4 means L1.

Oddly, writing to link_state is inconsistent and gets translated, leading
to mysterious results in which the value you store isn't comparable the
value you load back out.

Fix it by making link_state_store() match link_state_show().

[bhelgaas: Check "aspm_disabled" *before* validating input.  When
"aspm_disabled" is set, this changes the error for invalid input from
-EINVAL to -EPERM.]
Signed-off-by: NAndy Lutomirski <luto@kernel.org>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

AER errors might be recorded when powering-on devices.  These errors can be
ignored, so firmware usually clears them before the OS enumerates devices.
However, firmware is not involved when devices are added via hotplug, so
the OS may discover power-up errors that should be ignored.  The same may
happen when powering up devices when resuming after suspend.

Clear the AER error status registers during enumeration and resume.

[bhelgaas: changelog, remove repetitive comments]
Signed-off-by: NTaku Izumi <izumi.taku@jp.fujitsu.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

The function comment claimed this was pcie_port_device_suspend(), but it's
really pcie_port_device_resume().  Perils of cut and paste.

Use the correct function name in the comment.
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Acked-by: NRafael J. Wysocki <rafael.j.wysocki@intel.com>

Update the Link Control Enable Clock Power Management bit the same
way we update the ASPM Control bits, with a single call of
pcie_capability_clear_and_set_word().

No functional change; this just makes both paths use the same style.
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

Previously we assumed that PCIe Root Ports and Downstream Ports had Links
on their secondary side.  That is true in most systems, but it is possible
to connect a switch with either an Upstream or a Downstream Port leading
downstream.

Instead of relying on the component type to identify devices that have
links leading downstream, use the "dev->has_secondary_link" field.

[bhelgaas: changelog]
Signed-off-by: NYijing Wang <wangyijing@huawei.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

We allocate pcie_link_state for the component at the upstream end of a
Link.  Previously we did this by allocating pcie_link_state for Root Ports
and Downstream Ports.  This works fine for the typical topology:

  00:1c.0 Root Port       [bridge to bus 02]
  02:00.0 Upstream Port   [bridge to bus 03]
  03:00.0 Downstream Port [bridge to bus 04]
  04:00.0 Endpoint or Switch Port

However, it is possible to have a Root Port connected to a Downstream Port
instead of an Upstream Port, as in Robert White's ATCA system:

  00:1c.0 Root Port       [bridge to bus 02]
  02:00.0 Downstream Port [bridge to bus 03]
  03:01.0 Downstream Port [bridge to bus 04]
  04:00.0 Endpoint or Switch Port

In this topology, we wrongly allocated pcie_link_state for the 02:00.0
Downstream Port, which is actually the *downstream* end of a link.  This
led to the following NULL pointer dereference when we tried to connect this
link into the tree of links starting at the 00:1c.0 Root Port:

  BUG: unable to handle kernel NULL pointer dereference at 0000000000000088
  IP: [<ffffffff81550324>] pcie_aspm_init_link_state+0x744/0x850
  Hardware name: Kontron B3001/B3001, BIOS 4.6.3 08/07/2012
  Call Trace:
   [<ffffffff8153b865>] pci_scan_slot+0xd5/0x120
   [<ffffffff8153ca1d>] pci_scan_child_bus+0x2d/0xd0
   ...

Instead of relying on the component type to identify the upstream end of a
link, use the "dev->has_secondary_link" field.

This means it's now possible for an Upstream Port to have a link on its
secondary side, so alloc_pcie_link_state() needs to connect links
originating from both Upstream and Downstream Ports into the tree.

[bhelgaas: changelog, add comment]
Link: https://bugzilla.kernel.org/show_bug.cgi?id=94361
Link: http://lkml.kernel.org/r/54EB81B2.4050904@pobox.comReported-by: NRobert White <rwhite@pobox.com>
Signed-off-by: NYijing Wang <wangyijing@huawei.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

We decide in alloc_pcie_link_state() whether to allocate a pcie_link_state
for a device.  After that, it's sufficient to check pdev->link_state.  We
don't need to check the PCIe port type again.

Remove the redundant PCIe port type checking.

[bhelgaas: changelog]
Signed-off-by: NYijing Wang <wangyijing@huawei.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

After 387d3757 ("PCI: Don't clear ASPM bits when the FADT declares it's
unsupported"), the "force" parameter to __pci_disable_link_state() is
always "false".

Remove the "force" parameter and assume it's always false.
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

Communications with a hardware vendor confirm that the expected behaviour
on systems that set the FADT ASPM disable bit but which still grant full
PCIe control is for the OS to leave any BIOS configuration intact and
refuse to touch the ASPM bits.  This mimics the behaviour of Windows.
Signed-off-by: NMatthew Garrett <mjg59@coreos.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>

Commit fab4c256 ("PCI/AER: Add a TLP header print helper") introduced
the helper function __print_tlp_header(), but contrary to the intention,
the behaviour did change: Since we're taking the address of the parameter
t, the first 4 or 8 bytes printed will be the value of the pointer t
itself, and the remaining 12 or 8 bytes will be who-knows-what (something
from the stack).

We want to show the values of the four members of the struct
aer_header_log_regs; that can be done without ugly and error-prone casts.
On little-endian this should produce the same output as originally
intended, and since no-one has complained about getting garbage output so
far, I think big-endian should be ok too.

Fixes: fab4c256 ("PCI/AER: Add a TLP header print helper")
Signed-off-by: NRasmus Villemoes <linux@rasmusvillemoes.dk>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>
Acked-by: NBorislav Petkov <bp@suse.de>
CC: stable@vger.kernel.org	# v3.14+

The functions link_state_store() and clk_ctl_store() had just subtracted
ASCII '0' from input which could lead to undesired results.  Instead, use
Linux string functions to safely parse input.

[bhelgaas: check kstrtouint() return value]
Signed-off-by: NChris J Arges <chris.j.arges@canonical.com>
Signed-off-by: NBjorn Helgaas <bhelgaas@google.com>